Oct 30 21:39:36.108598 ip-10-0-130-60 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Oct 30 21:39:36.108606 ip-10-0-130-60 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Oct 30 21:39:36.108613 ip-10-0-130-60 systemd[1]: kubelet.service: Failed with result 'resources'.
Oct 30 21:39:36.108839 ip-10-0-130-60 systemd[1]: Failed to start Kubernetes Kubelet.
Oct 30 21:39:46.306490 ip-10-0-130-60 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Oct 30 21:39:46.306529 ip-10-0-130-60 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot db6d970df62647d394e18b765db49ded --
Oct 30 21:41:43.330999 ip-10-0-130-60 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 21:41:43.700353 ip-10-0-130-60 kubenswrapper[2580]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 21:41:43.700353 ip-10-0-130-60 kubenswrapper[2580]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 21:41:43.700353 ip-10-0-130-60 kubenswrapper[2580]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 21:41:43.700353 ip-10-0-130-60 kubenswrapper[2580]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 21:41:43.704374 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.704299    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:41:43.704374 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.704373    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:41:43.704374 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.704377    2580 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:41:43.704476 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.704380    2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:41:43.704476 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.704383    2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:41:43.704476 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.704386    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:41:43.704476 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.704388    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:41:43.704476 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.704391    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:41:43.704476 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.704394    2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:41:43.704476 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.704396    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:41:43.704476 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.704399    2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:41:43.704476 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.704402    2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:41:43.704476 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.704404    2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:41:43.704476 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.704407    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:41:43.704476 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.704410    2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:41:43.704476 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.704413    2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:41:43.704476 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.704415    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:41:43.704476 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.704418    2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:41:43.704476 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.704421    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:41:43.704476 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.704423    2580 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:41:43.704476 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.704426    2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:41:43.704476 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.704429    2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:41:43.704476 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.704439    2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:41:43.704953 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.704442    2580 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:41:43.704953 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.704445    2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:41:43.704953 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.704456    2580 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:41:43.704953 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.704458    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:41:43.704953 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.704461    2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:41:43.704953 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.704463    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:41:43.704953 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.704466    2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:41:43.704953 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.704469    2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:41:43.704953 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.704471    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:41:43.704953 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.704474    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:41:43.704953 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.704478    2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:41:43.704953 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.704483    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:41:43.704953 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.704487    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:41:43.704953 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.704490    2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:41:43.704953 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.704493    2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:41:43.704953 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.704496    2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:41:43.704953 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.704499    2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:41:43.704953 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.704501    2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:41:43.704953 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.704503    2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:41:43.705402 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.704506    2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:41:43.705402 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.704508    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:41:43.705402 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.704511    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:41:43.705402 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.704514    2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:41:43.705402 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.704516    2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:41:43.705402 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.704519    2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:41:43.705402 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.704521    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:41:43.705402 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.704523    2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:41:43.705402 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.704526    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:41:43.705402 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.704528    2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:41:43.705402 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.704531    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:41:43.705402 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.704533    2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:41:43.705402 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.704536    2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:41:43.705402 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.704544    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:41:43.705402 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.704548    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:41:43.705402 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.704551    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:41:43.705402 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.704554    2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:41:43.705402 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.704556    2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:41:43.705402 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.704559    2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:41:43.705866 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.704561    2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:41:43.705866 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.704564    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:41:43.705866 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.704566    2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:41:43.705866 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.704569    2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:41:43.705866 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.704573    2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:41:43.705866 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.704578    2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:41:43.705866 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.704581    2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:41:43.705866 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.704583    2580 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:41:43.705866 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.704586    2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:41:43.705866 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.704588    2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:41:43.705866 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.704591    2580 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:41:43.705866 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.704594    2580 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:41:43.705866 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.704596    2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:41:43.705866 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.704599    2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:41:43.705866 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.704601    2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:41:43.705866 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.704604    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:41:43.705866 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.704606    2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:41:43.705866 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.704609    2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:41:43.705866 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.704611    2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:41:43.706312 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.704614    2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:41:43.706312 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.704616    2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:41:43.706312 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.704619    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:41:43.706312 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.704621    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:41:43.706312 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.704624    2580 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:41:43.706312 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.704626    2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:41:43.706312 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.704629    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:41:43.706312 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.704631    2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:41:43.706312 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.704642    2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:41:43.706312 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.704645    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:41:43.706312 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.704648    2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:41:43.706312 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.704650    2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:41:43.706312 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.704653    2580 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:41:43.706312 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.704656    2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:41:43.706312 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.704658    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:41:43.706312 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.704661    2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:41:43.706312 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.704663    2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:41:43.706312 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.704666    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:41:43.706312 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.704668    2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:41:43.706312 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.704671    2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:41:43.706795 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.704673    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:41:43.706795 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.704676    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:41:43.706795 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.704678    2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:41:43.706795 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.704693    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:41:43.706795 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.704695    2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:41:43.706795 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.704699    2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:41:43.706795 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.705472    2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:41:43.706795 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.705479    2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:41:43.706795 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.705483    2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:41:43.706795 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.705485    2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:41:43.706795 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.705488    2580 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:41:43.706795 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.705491    2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:41:43.706795 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.705494    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:41:43.706795 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.705497    2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:41:43.706795 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.705502    2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:41:43.706795 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.705506    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:41:43.706795 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.705510    2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:41:43.706795 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.705514    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:41:43.706795 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.705517    2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:41:43.706795 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.705519    2580 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:41:43.707270 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.705524    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:41:43.707270 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.705528    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:41:43.707270 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.705530    2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:41:43.707270 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.705533    2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:41:43.707270 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.705536    2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:41:43.707270 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.705538    2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:41:43.707270 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.705541    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:41:43.707270 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.705544    2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:41:43.707270 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.705546    2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:41:43.707270 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.705549    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:41:43.707270 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.705551    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:41:43.707270 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.705554    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:41:43.707270 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.705556    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:41:43.707270 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.705559    2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:41:43.707270 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.705561    2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:41:43.707270 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.705564    2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:41:43.707270 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.705567    2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:41:43.707270 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.705570    2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:41:43.707270 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.705572    2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:41:43.707270 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.705575    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:41:43.707764 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.705578    2580 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:41:43.707764 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.705580    2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:41:43.707764 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.705583    2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:41:43.707764 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.705585    2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:41:43.707764 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.705588    2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:41:43.707764 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.705590    2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:41:43.707764 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.705593    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:41:43.707764 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.705595    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:41:43.707764 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.705598    2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:41:43.707764 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.705600    2580 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:41:43.707764 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.705603    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:41:43.707764 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.705606    2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:41:43.707764 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.705609    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:41:43.707764 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.705611    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:41:43.707764 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.705614    2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:41:43.707764 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.705617    2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:41:43.707764 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.705620    2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:41:43.707764 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.705622    2580 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:41:43.707764 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.705625    2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:41:43.707764 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.705627    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:41:43.708239 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.705630    2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:41:43.708239 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.705632    2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:41:43.708239 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.705635    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:41:43.708239 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.705637    2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:41:43.708239 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.705640    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:41:43.708239 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.705642    2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:41:43.708239 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.705645    2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:41:43.708239 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.705648    2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:41:43.708239 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.705650    2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:41:43.708239 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.705653    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:41:43.708239 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.705656    2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:41:43.708239 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.705658    2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:41:43.708239 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.705661    2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:41:43.708239 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.705663    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:41:43.708239 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.705666    2580 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:41:43.708239 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.705669    2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:41:43.708239 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.705672    2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:41:43.708239 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.705674    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:41:43.708239 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.705678    2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:41:43.708708 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.705697    2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:41:43.708708 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.705700    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:41:43.708708 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.705703    2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:41:43.708708 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.705706    2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:41:43.708708 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.705709    2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:41:43.708708 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.705712    2580 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:41:43.708708 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.705716    2580 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:41:43.708708 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.705721    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:41:43.708708 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.705725    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:41:43.708708 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.705730    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:41:43.708708 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.705733    2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:41:43.708708 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.705737    2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:41:43.708708 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.705741    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:41:43.708708 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.705746    2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:41:43.708708 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.705749    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:41:43.708708 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.705752    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:41:43.708708 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.705754    2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:41:43.708708 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.705757    2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:41:43.708708 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.705760    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:41:43.709185 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.705763    2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:41:43.709185 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.705765    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:41:43.709185 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.705768    2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:41:43.709185 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.705770    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:41:43.709185 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.705773    2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:41:43.709185 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.705775    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:41:43.709185 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.705778    2580 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:41:43.709185 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.705782    2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:41:43.709185 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.705784    2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:41:43.709185 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.705787    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:41:43.709185 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.705790    2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:41:43.709185 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.705793    2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:41:43.709185 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.705795    2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:41:43.709185 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.705798    2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:41:43.709185 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.706271    2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:41:43.709185 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.706276    2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:41:43.709185 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.706278    2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:41:43.709185 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.706281    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:41:43.709185 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.706284    2580 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:41:43.709185 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.706287    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:41:43.709652 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.706290    2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:41:43.709652 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.706293    2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:41:43.709652 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.706296    2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:41:43.709652 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.706301    2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:41:43.709652 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.706304    2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:41:43.709652 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.706307    2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:41:43.709652 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.706309    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:41:43.709652 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.706312    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:41:43.709652 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.706315    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:41:43.709652 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.706317    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:41:43.709652 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.706320    2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:41:43.709652 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.706323    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:41:43.709652 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.706326    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:41:43.709652 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.706329    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:41:43.709652 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.706331    2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:41:43.709652 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.706334    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:41:43.709652 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.706336    2580 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:41:43.709652 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.706339    2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:41:43.709652 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.706341    2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:41:43.710130 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.706344    2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:41:43.710130 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.706347    2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:41:43.710130 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.706349    2580 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:41:43.710130 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.706353    2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:41:43.710130 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.706356    2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:41:43.710130 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.706358    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:41:43.710130 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.706361    2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:41:43.710130 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.706363    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:41:43.710130 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.706366    2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:41:43.710130 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.706369    2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:41:43.710130 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.706371    2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:41:43.710130 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.706374    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:41:43.710130 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.706376    2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:41:43.710130 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.706379    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:41:43.710130 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.706382    2580 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:41:43.710130 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.706385    2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:41:43.710130 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.706389    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:41:43.710130 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.706392    2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:41:43.710130 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.706395    2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:41:43.710607 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.706399    2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:41:43.710607 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.706403    2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:41:43.710607 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.706406    2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:41:43.710607 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.706409    2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:41:43.710607 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.706411    2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:41:43.710607 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.706414    2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:41:43.710607 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.706417    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:41:43.710607 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.706420    2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:41:43.710607 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.706422    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:41:43.710607 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.706425    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:41:43.710607 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.706427    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:41:43.710607 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.706430    2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:41:43.710607 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.706432    2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:41:43.710607 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.706435    2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:41:43.710607 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.706437    2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:41:43.710607 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.706440    2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:41:43.710607 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.706444    2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:41:43.710607 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.706447    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:41:43.710607 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.706449    2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:41:43.711084 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.706452    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:41:43.711084 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.706455    2580 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:41:43.711084 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.706457    2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:41:43.711084 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.706459    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:41:43.711084 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.706462    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:41:43.711084 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.706465    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:41:43.711084 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.706467    2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:41:43.711084 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.706470    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:41:43.711084 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.706472    2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:41:43.711084 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.706475    2580 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:41:43.711084 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.706480    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:41:43.711084 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.706482    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:41:43.711084 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.706485    2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:41:43.711084 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.706487    2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:41:43.711084 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.706490    2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:41:43.711084 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.706492    2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:41:43.711084 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.706495    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:41:43.711084 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.706497    2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:41:43.711084 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.706500    2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:41:43.711084 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.706502    2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:41:43.711554 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.706505    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:41:43.711554 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.706507    2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:41:43.711554 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.706510    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:41:43.711554 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.706513    2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:41:43.711554 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.706517    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:41:43.711554 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.706520    2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:41:43.711554 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.706523    2580 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:41:43.711554 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.706525    2580 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:41:43.711554 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.706528    2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:41:43.711554 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.706531    2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:41:43.711554 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.706534    2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:41:43.711554 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.706536    2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:41:43.711554 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.706539    2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:41:43.711554 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.706543    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:41:43.711554 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.706545    2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:41:43.711554 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.706548    2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:41:43.711554 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.706550    2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:41:43.711554 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.706553    2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:41:43.711554 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.706556    2580 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:41:43.711554 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.706558    2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:41:43.712075 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.706560    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:41:43.712075 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.706563    2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:41:43.712075 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.706567    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:41:43.712075 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708190    2580 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 21:41:43.712075 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708201    2580 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 21:41:43.712075 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708208    2580 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 21:41:43.712075 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708212    2580 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 21:41:43.712075 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708217    2580 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 21:41:43.712075 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708221    2580 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 21:41:43.712075 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708226    2580 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 21:41:43.712075 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708229    2580 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 21:41:43.712075 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708233    2580 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 21:41:43.712075 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708236    2580 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 21:41:43.712075 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708240    2580 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 21:41:43.712075 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708244    2580 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 21:41:43.712075 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708247    2580 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 21:41:43.712075 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708250    2580 flags.go:64] FLAG: --cgroup-root=""
Oct 30 21:41:43.712075 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708253    2580 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 21:41:43.712075 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708256    2580 flags.go:64] FLAG: --client-ca-file=""
Oct 30 21:41:43.712075 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708259    2580 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 21:41:43.712075 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708262    2580 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 21:41:43.712075 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708267    2580 flags.go:64] FLAG: --cluster-domain=""
Oct 30 21:41:43.712075 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708270    2580 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 21:41:43.712625 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708273    2580 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 21:41:43.712625 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708276    2580 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 21:41:43.712625 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708280    2580 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 21:41:43.712625 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708283    2580 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 21:41:43.712625 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708286    2580 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 21:41:43.712625 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708290    2580 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 21:41:43.712625 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708293    2580 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 21:41:43.712625 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708296    2580 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 21:41:43.712625 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708299    2580 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 21:41:43.712625 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708301    2580 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 21:41:43.712625 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708305    2580 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 21:41:43.712625 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708308    2580 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 21:41:43.712625 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708312    2580 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 21:41:43.712625 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708316    2580 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 21:41:43.712625 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708319    2580 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 21:41:43.712625 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708322    2580 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 21:41:43.712625 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708325    2580 flags.go:64] FLAG: --enable-server="true"
Oct 30 21:41:43.712625 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708328    2580 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 21:41:43.712625 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708332    2580 flags.go:64] FLAG: --event-burst="100"
Oct 30 21:41:43.712625 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708335    2580 flags.go:64] FLAG: --event-qps="50"
Oct 30 21:41:43.712625 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708338    2580 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 21:41:43.712625 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708341    2580 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 21:41:43.712625 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708344    2580 flags.go:64] FLAG: --eviction-hard=""
Oct 30 21:41:43.712625 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708348    2580 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 21:41:43.713216 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708351    2580 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 21:41:43.713216 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708354    2580 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 21:41:43.713216 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708357    2580 flags.go:64] FLAG: --eviction-soft=""
Oct 30 21:41:43.713216 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708360    2580 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 21:41:43.713216 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708363    2580 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 21:41:43.713216 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708366    2580 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 21:41:43.713216 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708369    2580 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 21:41:43.713216 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708373    2580 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 21:41:43.713216 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708376    2580 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 21:41:43.713216 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708379    2580 flags.go:64] FLAG: --feature-gates=""
Oct 30 21:41:43.713216 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708383    2580 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 21:41:43.713216 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708386    2580 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 21:41:43.713216 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708389    2580 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 21:41:43.713216 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708392    2580 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 21:41:43.713216 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708396    2580 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 21:41:43.713216 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708399    2580 flags.go:64] FLAG: --help="false"
Oct 30 21:41:43.713216 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708402    2580 flags.go:64] FLAG: --hostname-override="ip-10-0-130-60.us-west-2.compute.internal"
Oct 30 21:41:43.713216 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708405    2580 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 21:41:43.713216 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708408    2580 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 21:41:43.713216 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708411    2580 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 21:41:43.713216 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708414    2580 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 21:41:43.713216 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708418    2580 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 21:41:43.713216 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708421    2580 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 21:41:43.713832 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708424    2580 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 21:41:43.713832 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708427    2580 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 21:41:43.713832 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708430    2580 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 21:41:43.713832 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708433    2580 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 21:41:43.713832 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708436    2580 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 21:41:43.713832 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708439    2580 flags.go:64] FLAG: --kube-reserved=""
Oct 30 21:41:43.713832 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708442    2580 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 21:41:43.713832 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708444    2580 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 21:41:43.713832 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708454    2580 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 21:41:43.713832 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708458    2580 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 21:41:43.713832 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708461    2580 flags.go:64] FLAG: --lock-file=""
Oct 30 21:41:43.713832 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708464    2580 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 21:41:43.713832 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708467    2580 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 21:41:43.713832 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708470    2580 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 21:41:43.713832 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708475    2580 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 21:41:43.713832 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708479    2580 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 21:41:43.713832 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708482    2580 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 21:41:43.713832 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708484    2580 flags.go:64] FLAG: --logging-format="text"
Oct 30 21:41:43.713832 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708487    2580 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 21:41:43.713832 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708491    2580 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 21:41:43.713832 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708493    2580 flags.go:64] FLAG: --manifest-url=""
Oct 30 21:41:43.713832 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708496    2580 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 21:41:43.713832 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708501    2580 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 21:41:43.713832 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708504    2580 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 21:41:43.713832 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708507    2580 flags.go:64] FLAG: --max-pods="110"
Oct 30 21:41:43.714471 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708510    2580 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 21:41:43.714471 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708513    2580 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 21:41:43.714471 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708516    2580 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 21:41:43.714471 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708519    2580 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 21:41:43.714471 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708522    2580 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 21:41:43.714471 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708525    2580 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 21:41:43.714471 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708528    2580 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 21:41:43.714471 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708535    2580 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 21:41:43.714471 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708539    2580 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 21:41:43.714471 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708541    2580 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 21:41:43.714471 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708544    2580 flags.go:64] FLAG: --pod-cidr=""
Oct 30 21:41:43.714471 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708547    2580 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 21:41:43.714471 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708550    2580 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 21:41:43.714471 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708553    2580 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 21:41:43.714471 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708556    2580 flags.go:64] FLAG: --port="10250"
Oct 30 21:41:43.714471 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708559    2580 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 21:41:43.714471 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708562    2580 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-08205b8e8f123ed6b"
Oct 30 21:41:43.714471 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708565    2580 flags.go:64] FLAG: --qos-reserved=""
Oct 30 21:41:43.714471 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708568    2580 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 21:41:43.714471 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708571    2580 flags.go:64] FLAG: --register-node="true"
Oct 30 21:41:43.714471 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708574    2580 flags.go:64] FLAG: --register-with-taints=""
Oct 30 21:41:43.714471 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708577    2580 flags.go:64] FLAG: --registry-burst="10"
Oct 30 21:41:43.714471 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708580    2580 flags.go:64] FLAG: --registry-qps="5"
Oct 30 21:41:43.714471 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708583    2580 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 21:41:43.714471 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708587    2580 flags.go:64] FLAG: --reserved-memory=""
Oct 30 21:41:43.714471 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708591    2580 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 21:41:43.715611 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708594    2580 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 21:41:43.715611 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708597    2580 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 21:41:43.715611 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708600    2580 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 21:41:43.715611 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708603    2580 flags.go:64] FLAG: --runonce="false"
Oct 30 21:41:43.715611 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708606    2580 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 21:41:43.715611 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708609    2580 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 21:41:43.715611 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708612    2580 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 21:41:43.715611 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708615    2580 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 21:41:43.715611 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708617    2580 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 21:41:43.715611 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708620    2580 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 21:41:43.715611 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708623    2580 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 21:41:43.715611 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708627    2580 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 21:41:43.715611 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708630    2580 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 21:41:43.715611 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708633    2580 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 21:41:43.715611 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708636    2580 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 21:41:43.715611 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708639    2580 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 21:41:43.715611 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708642    2580 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 21:41:43.715611 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708645    2580 flags.go:64] FLAG: --system-cgroups=""
Oct 30 21:41:43.715611 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708648    2580 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 21:41:43.715611 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708654    2580 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 21:41:43.715611 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708656    2580 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 21:41:43.715611 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708659    2580 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 21:41:43.715611 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708662    2580 flags.go:64] FLAG: --tls-min-version=""
Oct 30 21:41:43.715611 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708665    2580 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 21:41:43.715611 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708670    2580 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 21:41:43.716763 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708673    2580 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 21:41:43.716763 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708676    2580 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 21:41:43.716763 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708688    2580 flags.go:64] FLAG: --v="2"
Oct 30 21:41:43.716763 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708694    2580 flags.go:64] FLAG: --version="false"
Oct 30 21:41:43.716763 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708698    2580 flags.go:64] FLAG: --vmodule=""
Oct 30 21:41:43.716763 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708702    2580 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 21:41:43.716763 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.708706    2580 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 21:41:43.716763 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.708841    2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:41:43.716763 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.708845    2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:41:43.716763 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.708847    2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:41:43.716763 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.708850    2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:41:43.716763 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.708853    2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:41:43.716763 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.708855    2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:41:43.716763 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.708858    2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:41:43.716763 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.708861    2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:41:43.716763 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.708863    2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:41:43.716763 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.708866    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:41:43.716763 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.708868    2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:41:43.716763 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.708871    2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:41:43.716763 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.708873    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:41:43.716763 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.708876    2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:41:43.717625 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.708879    2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:41:43.717625 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.708882    2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:41:43.717625 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.708884    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:41:43.717625 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.708887    2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:41:43.717625 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.708890    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:41:43.717625 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.708892    2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:41:43.717625 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.708895    2580 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:41:43.717625 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.708897    2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:41:43.717625 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.708900    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:41:43.717625 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.708902    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:41:43.717625 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.708906    2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:41:43.717625 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.708909    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:41:43.717625 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.708912    2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:41:43.717625 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.708914    2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:41:43.717625 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.708918    2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:41:43.717625 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.708921    2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:41:43.717625 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.708924    2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:41:43.717625 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.708927    2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:41:43.717625 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.708929    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:41:43.717625 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.708932    2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:41:43.718259 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.708935    2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:41:43.718259 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.708937    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:41:43.718259 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.708940    2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:41:43.718259 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.708943    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:41:43.718259 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.708945    2580 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:41:43.718259 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.708948    2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:41:43.718259 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.708950    2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:41:43.718259 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.708953    2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:41:43.718259 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.708955    2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:41:43.718259 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.708958    2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:41:43.718259 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.708961    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:41:43.718259 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.708963    2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:41:43.718259 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.708966    2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:41:43.718259 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.708969    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:41:43.718259 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.708972    2580 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:41:43.718259 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.708974    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:41:43.718259 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.708977    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:41:43.718259 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.708979    2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:41:43.718259 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.708982    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:41:43.718259 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.708984    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:41:43.718763 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.708987    2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:41:43.718763 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.708989    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:41:43.718763 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.708993    2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:41:43.718763 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.708996    2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:41:43.718763 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709000    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:41:43.718763 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709002    2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:41:43.718763 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709006    2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:41:43.718763 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709009    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:41:43.718763 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709011    2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:41:43.718763 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709014    2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:41:43.718763 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709017    2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:41:43.718763 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709019    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:41:43.718763 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709022    2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:41:43.718763 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709024    2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:41:43.718763 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709027    2580 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:41:43.718763 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709030    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:41:43.718763 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709033    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:41:43.718763 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709035    2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:41:43.718763 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709037    2580 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:41:43.719225 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709040    2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:41:43.719225 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709042    2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:41:43.719225 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709045    2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:41:43.719225 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709047    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:41:43.719225 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709050    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:41:43.719225 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709052    2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:41:43.719225 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709055    2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:41:43.719225 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709059    2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:41:43.719225 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709063    2580 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:41:43.719225 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709066    2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:41:43.719225 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709069    2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:41:43.719225 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709072    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:41:43.719225 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709074    2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:41:43.719225 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709077    2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:41:43.719225 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709079    2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:41:43.719225 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709083    2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:41:43.719225 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709085    2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:41:43.719225 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709088    2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:41:43.719225 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709090    2580 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:41:43.719669 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709096    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:41:43.719669 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709098    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:41:43.719669 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709101    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:41:43.719669 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709104    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:41:43.719669 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709106    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:41:43.719669 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709109    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:41:43.719669 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709112    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:41:43.719669 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709114    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:41:43.719669 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709117    2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:41:43.719669 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709120    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:41:43.719669 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709122    2580 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:41:43.719669 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709125    2580 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:41:43.719669 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709127    2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:41:43.719669 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709130    2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:41:43.719669 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.709155    2580 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 21:41:43.719669 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709190    2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:41:43.720077 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709195    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:41:43.720077 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709198    2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:41:43.720077 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709201    2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:41:43.720077 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709205    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:41:43.720077 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709207    2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:41:43.720077 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709210    2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:41:43.720077 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709213    2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:41:43.720077 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709216    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:41:43.720077 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709218    2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:41:43.720077 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709221    2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:41:43.720077 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709223    2580 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:41:43.720077 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709226    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:41:43.720077 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709230    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:41:43.720077 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709233    2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:41:43.720077 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709235    2580 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:41:43.720077 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709238    2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:41:43.720077 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709240    2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:41:43.720077 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709244    2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:41:43.720077 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709247    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:41:43.720077 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709249    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:41:43.720566 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709252    2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:41:43.720566 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709254    2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:41:43.720566 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709258    2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:41:43.720566 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709262    2580 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:41:43.720566 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709265    2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:41:43.720566 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709268    2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:41:43.720566 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709271    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:41:43.720566 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709274    2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:41:43.720566 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709276    2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:41:43.720566 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709279    2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:41:43.720566 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709281    2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:41:43.720566 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709284    2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:41:43.720566 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709286    2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:41:43.720566 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709289    2580 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:41:43.720566 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709292    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:41:43.720566 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709294    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:41:43.720566 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709297    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:41:43.720566 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709299    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:41:43.720566 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709302    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:41:43.721041 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709306    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:41:43.721041 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709309    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:41:43.721041 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709311    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:41:43.721041 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709314    2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:41:43.721041 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709317    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:41:43.721041 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709320    2580 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:41:43.721041 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709323    2580 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:41:43.721041 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709325    2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:41:43.721041 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709328    2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:41:43.721041 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709330    2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:41:43.721041 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709334    2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:41:43.721041 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709337    2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:41:43.721041 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709340    2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:41:43.721041 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709343    2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:41:43.721041 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709345    2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:41:43.721041 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709348    2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:41:43.721041 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709351    2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:41:43.721041 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709354    2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:41:43.721041 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709356    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:41:43.721041 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709359    2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:41:43.721511 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709361    2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:41:43.721511 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709364    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:41:43.721511 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709367    2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:41:43.721511 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709369    2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:41:43.721511 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709372    2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:41:43.721511 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709374    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:41:43.721511 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709377    2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:41:43.721511 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709380    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:41:43.721511 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709382    2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:41:43.721511 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709385    2580 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:41:43.721511 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709388    2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:41:43.721511 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709390    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:41:43.721511 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709392    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:41:43.721511 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709395    2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:41:43.721511 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709398    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:41:43.721511 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709401    2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:41:43.721511 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709403    2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:41:43.721511 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709407    2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:41:43.721511 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709409    2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:41:43.721511 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709412    2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:41:43.721992 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709414    2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:41:43.721992 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709417    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:41:43.721992 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709421    2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:41:43.721992 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709424    2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:41:43.721992 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709426    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:41:43.721992 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709429    2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:41:43.721992 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709432    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:41:43.721992 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709434    2580 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:41:43.721992 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709437    2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:41:43.721992 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709440    2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:41:43.721992 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709442    2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:41:43.721992 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709445    2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:41:43.721992 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709447    2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:41:43.721992 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709450    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:41:43.721992 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709452    2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:41:43.721992 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709455    2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:41:43.721992 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709458    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:41:43.721992 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709460    2580 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:41:43.721992 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709462    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:41:43.721992 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709465    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:41:43.722490 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709468    2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:41:43.722490 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709470    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:41:43.722490 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709474    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:41:43.722490 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709476    2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:41:43.722490 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709479    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:41:43.722490 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.709481    2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:41:43.722490 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.718585    2580 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 21:41:43.722490 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.718602    2580 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 21:41:43.722490 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.718624    2580 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 21:41:43.722490 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.718630    2580 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 21:41:43.722490 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.718705    2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:41:43.722490 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.718711    2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:41:43.722490 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.718714    2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:41:43.722490 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.718718    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:41:43.722490 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.718721    2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:41:43.722490 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.718724    2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:41:43.722490 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.718727    2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:41:43.722490 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.718730    2580 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:41:43.722490 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.718733    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:41:43.723031 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.718737    2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:41:43.723031 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.718741    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:41:43.723031 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.718744    2580 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:41:43.723031 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.718746    2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:41:43.723031 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.718749    2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:41:43.723031 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.718752    2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:41:43.723031 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.718755    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:41:43.723031 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.718757    2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:41:43.723031 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.718760    2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:41:43.723031 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.718763    2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:41:43.723031 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.718765    2580 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:41:43.723031 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.718774    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:41:43.723031 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.718777    2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:41:43.723031 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.718780    2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:41:43.723031 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.718782    2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:41:43.723031 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.718785    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:41:43.723031 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.718788    2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:41:43.723031 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.718790    2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:41:43.723031 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.718793    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:41:43.723495 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.718796    2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:41:43.723495 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.718799    2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:41:43.723495 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.718802    2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:41:43.723495 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.718805    2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:41:43.723495 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.718808    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:41:43.723495 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.718810    2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:41:43.723495 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.718813    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:41:43.723495 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.718815    2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:41:43.723495 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.718818    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:41:43.723495 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.718821    2580 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:41:43.723495 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.718823    2580 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:41:43.723495 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.718826    2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:41:43.723495 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.718828    2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:41:43.723495 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.718831    2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:41:43.723495 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.718833    2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:41:43.723495 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.718836    2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:41:43.723495 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.718838    2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:41:43.723495 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.718841    2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:41:43.723495 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.718844    2580 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:41:43.723495 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.718846    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:41:43.724007 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.718849    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:41:43.724007 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.718852    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:41:43.724007 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.718855    2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:41:43.724007 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.718857    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:41:43.724007 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.718860    2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:41:43.724007 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.718863    2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:41:43.724007 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.718866    2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:41:43.724007 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.718869    2580 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:41:43.724007 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.718872    2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:41:43.724007 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.718874    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:41:43.724007 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.718876    2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:41:43.724007 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.718879    2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:41:43.724007 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.718882    2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:41:43.724007 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.718885    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:41:43.724007 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.718887    2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:41:43.724007 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.718890    2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:41:43.724007 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.718893    2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:41:43.724007 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.718895    2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:41:43.724007 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.718898    2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:41:43.724007 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.718900    2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:41:43.724468 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.718903    2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:41:43.724468 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.718905    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:41:43.724468 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.718908    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:41:43.724468 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.718910    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:41:43.724468 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.718913    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:41:43.724468 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.718915    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:41:43.724468 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.718918    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:41:43.724468 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.718920    2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:41:43.724468 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.718923    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:41:43.724468 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.718925    2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:41:43.724468 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.718928    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:41:43.724468 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.718931    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:41:43.724468 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.718934    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:41:43.724468 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.718937    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:41:43.724468 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.718939    2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:41:43.724468 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.718941    2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:41:43.724468 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.718946    2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:41:43.724468 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.718950    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:41:43.724468 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.718953    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:41:43.724928 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.718955    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:41:43.724928 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.718958    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:41:43.724928 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.718961    2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:41:43.724928 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.718963    2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:41:43.724928 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.718966    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:41:43.724928 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.718969    2580 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:41:43.724928 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.718971    2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:41:43.724928 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.718974    2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:41:43.724928 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.718977    2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:41:43.724928 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.718979    2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:41:43.724928 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.718982    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:41:43.724928 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.718985    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:41:43.724928 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.718987    2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:41:43.724928 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.718990    2580 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:41:43.724928 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.718992    2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:41:43.724928 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.718995    2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:41:43.724928 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.718998    2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:41:43.724928 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719000    2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:41:43.724928 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719003    2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:41:43.725383 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.719031    2580 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 21:41:43.725383 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719170    2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:41:43.725383 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719174    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:41:43.725383 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719178    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:41:43.725383 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719181    2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:41:43.725383 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719184    2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:41:43.725383 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719187    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:41:43.725383 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719190    2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:41:43.725383 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719192    2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:41:43.725383 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719195    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:41:43.725383 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719197    2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:41:43.725383 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719200    2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:41:43.725383 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719203    2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:41:43.725383 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719206    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:41:43.725383 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719208    2580 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:41:43.725383 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719211    2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:41:43.725850 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719213    2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:41:43.725850 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719216    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:41:43.725850 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719219    2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:41:43.725850 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719221    2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:41:43.725850 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719224    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:41:43.725850 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719226    2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:41:43.725850 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719229    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:41:43.725850 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719232    2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:41:43.725850 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719234    2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:41:43.725850 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719237    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:41:43.725850 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719240    2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:41:43.725850 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719242    2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:41:43.725850 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719245    2580 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:41:43.725850 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719247    2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:41:43.725850 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719249    2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:41:43.725850 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719252    2580 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:41:43.725850 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719254    2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:41:43.725850 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719257    2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:41:43.725850 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719260    2580 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:41:43.725850 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719262    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:41:43.726314 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719264    2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:41:43.726314 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719269    2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:41:43.726314 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719272    2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:41:43.726314 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719276    2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:41:43.726314 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719279    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:41:43.726314 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719282    2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:41:43.726314 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719285    2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:41:43.726314 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719288    2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:41:43.726314 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719290    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:41:43.726314 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719293    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:41:43.726314 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719295    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:41:43.726314 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719298    2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:41:43.726314 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719300    2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:41:43.726314 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719303    2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:41:43.726314 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719306    2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:41:43.726314 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719308    2580 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:41:43.726314 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719310    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:41:43.726314 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719313    2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:41:43.726314 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719316    2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:41:43.726843 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719318    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:41:43.726843 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719320    2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:41:43.726843 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719323    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:41:43.726843 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719326    2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:41:43.726843 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719328    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:41:43.726843 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719330    2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:41:43.726843 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719333    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:41:43.726843 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719335    2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:41:43.726843 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719338    2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:41:43.726843 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719340    2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:41:43.726843 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719343    2580 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:41:43.726843 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719345    2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:41:43.726843 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719348    2580 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:41:43.726843 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719350    2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:41:43.726843 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719353    2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:41:43.726843 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719355    2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:41:43.726843 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719358    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:41:43.726843 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719363    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:41:43.726843 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719366    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:41:43.726843 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719369    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:41:43.727358 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719374    2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:41:43.727358 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719376    2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:41:43.727358 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719379    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:41:43.727358 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719381    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:41:43.727358 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719384    2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:41:43.727358 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719387    2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:41:43.727358 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719389    2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:41:43.727358 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719392    2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:41:43.727358 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719394    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:41:43.727358 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719397    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:41:43.727358 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719400    2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:41:43.727358 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719402    2580 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:41:43.727358 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719406    2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:41:43.727358 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719409    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:41:43.727358 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719411    2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:41:43.727358 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719414    2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:41:43.727358 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719417    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:41:43.727358 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719419    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:41:43.727358 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719423    2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:41:43.727358 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719426    2580 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:41:43.727865 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719430    2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:41:43.727865 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719433    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:41:43.727865 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719436    2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:41:43.727865 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719439    2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:41:43.727865 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719442    2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:41:43.727865 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719444    2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:41:43.727865 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719447    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:41:43.727865 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719450    2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:41:43.727865 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719452    2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:41:43.727865 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719455    2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:41:43.727865 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719458    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:41:43.727865 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719461    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:41:43.727865 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.719485    2580 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 21:41:43.727865 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719517    2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:41:43.727865 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719521    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:41:43.727865 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719525    2580 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:41:43.728242 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719528    2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:41:43.728242 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719530    2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:41:43.728242 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719533    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:41:43.728242 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719535    2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:41:43.728242 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719538    2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:41:43.728242 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719540    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:41:43.728242 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719543    2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:41:43.728242 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719546    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:41:43.728242 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719550    2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:41:43.728242 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719553    2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:41:43.728242 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719555    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:41:43.728242 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719558    2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:41:43.728242 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719561    2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:41:43.728242 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719564    2580 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:41:43.728242 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719566    2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:41:43.728242 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719569    2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:41:43.728242 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719571    2580 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:41:43.728242 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719574    2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:41:43.728242 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719576    2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:41:43.728242 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719579    2580 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:41:43.728755 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719581    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:41:43.728755 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719584    2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:41:43.728755 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719588    2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:41:43.728755 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719592    2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:41:43.728755 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719594    2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:41:43.728755 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719598    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:41:43.728755 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719601    2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:41:43.728755 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719603    2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:41:43.728755 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719606    2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:41:43.728755 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719611    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:41:43.728755 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719613    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:41:43.728755 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719616    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:41:43.728755 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719619    2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:41:43.728755 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719621    2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:41:43.728755 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719624    2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:41:43.728755 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719626    2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:41:43.728755 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719629    2580 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:41:43.728755 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719632    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:41:43.728755 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719634    2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:41:43.729231 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719637    2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:41:43.729231 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719640    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:41:43.729231 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719643    2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:41:43.729231 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719645    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:41:43.729231 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719648    2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:41:43.729231 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719650    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:41:43.729231 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719653    2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:41:43.729231 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719656    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:41:43.729231 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719658    2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:41:43.729231 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719660    2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:41:43.729231 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719663    2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:41:43.729231 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719666    2580 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:41:43.729231 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719668    2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:41:43.729231 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719671    2580 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:41:43.729231 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719674    2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:41:43.729231 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719676    2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:41:43.729231 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719692    2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:41:43.729231 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719696    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:41:43.729231 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719698    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:41:43.729231 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719701    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:41:43.729735 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719704    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:41:43.729735 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719707    2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:41:43.729735 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719712    2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:41:43.729735 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719714    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:41:43.729735 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719717    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:41:43.729735 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719719    2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:41:43.729735 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719722    2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:41:43.729735 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719725    2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:41:43.729735 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719727    2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:41:43.729735 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719730    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:41:43.729735 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719732    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:41:43.729735 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719735    2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:41:43.729735 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719737    2580 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:41:43.729735 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719741    2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:41:43.729735 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719744    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:41:43.729735 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719746    2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:41:43.729735 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719750    2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:41:43.729735 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719752    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:41:43.729735 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719755    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:41:43.729735 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719759    2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:41:43.730203 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719762    2580 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:41:43.730203 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719765    2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:41:43.730203 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719768    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:41:43.730203 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719771    2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:41:43.730203 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719773    2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:41:43.730203 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719776    2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:41:43.730203 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719778    2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:41:43.730203 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719781    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:41:43.730203 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719784    2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:41:43.730203 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719786    2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:41:43.730203 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719789    2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:41:43.730203 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719791    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:41:43.730203 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719794    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:41:43.730203 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719797    2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:41:43.730203 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719801    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:41:43.730203 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719804    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:41:43.730203 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719807    2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:41:43.730203 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719809    2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:41:43.730203 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719813    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:41:43.730203 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719816    2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:41:43.730664 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719818    2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:41:43.730664 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719821    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:41:43.730664 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719823    2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:41:43.730664 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:43.719826    2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:41:43.730664 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.719890    2580 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 21:41:43.730664 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.722932    2580 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 30 21:41:43.730664 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.723798    2580 server.go:1014] "Starting client certificate rotation"
Oct 30 21:41:43.730664 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.723896    2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 21:41:43.730664 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.723929    2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 21:41:43.730664 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.725707    2580 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 21:41:43.730664 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.726828    2580 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 21:41:43.737400 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.737385    2580 log.go:25] "Validated CRI v1 runtime API"
Oct 30 21:41:43.740107 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.740094    2580 log.go:25] "Validated CRI v1 image API"
Oct 30 21:41:43.740915 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.740893    2580 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 21:41:43.743358 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.743333    2580 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 a2fd7be1-fe1e-4e94-93c3-74428fedeaca:/dev/nvme0n1p3 fb547cbe-efef-43c5-9742-090921dd19a8:/dev/nvme0n1p4]
Oct 30 21:41:43.743426 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.743356    2580 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 30 21:41:43.749641 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.749533    2580 manager.go:217] Machine: {Timestamp:2026-10-30 21:41:43.748295106 +0000 UTC m=+0.312470343 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098743 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2d41036d8f146a8fdfb95efa4ac9ad SystemUUID:ec2d4103-6d8f-146a-8fdf-b95efa4ac9ad BootID:db6d970d-f626-47d3-94e1-8b765db49ded Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:06:ff:e3:9e:ab:e3 Speed:0 Mtu:9001} {Name:ens5 MacAddress:06:ff:e3:9e:ab:e3 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:2e:d0:97:1b:9b:03 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 21:41:43.749641 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.749639    2580 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 21:41:43.749753 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.749727    2580 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 21:41:43.751294 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.751273    2580 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 21:41:43.751428 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.751297    2580 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-60.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 21:41:43.751468 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.751430    2580 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 21:41:43.751468 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.751438    2580 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 21:41:43.751523 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.751469    2580 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 21:41:43.751523 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.751486    2580 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 21:41:43.752491 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.752481    2580 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 21:41:43.756723 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.756711    2580 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 21:41:43.756842 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.756834    2580 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 21:41:43.757520 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.757512    2580 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 21:41:43.757551 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.757525    2580 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 21:41:43.757551 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.757536    2580 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 21:41:43.757551 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.757544    2580 kubelet.go:405] "Adding apiserver pod source"
Oct 30 21:41:43.757551 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.757552    2580 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 21:41:43.758083 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.758059    2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 21:41:43.760122 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.760106    2580 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 21:41:43.767017 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.767000    2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 21:41:43.767837 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.767823    2580 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 21:41:43.767879 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.767855    2580 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 21:41:43.768016 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.768005    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 21:41:43.768062 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.768022    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 21:41:43.768062 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.768028    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 21:41:43.768062 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.768034    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 21:41:43.768062 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.768040    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 21:41:43.768062 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.768049    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 21:41:43.768597 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.768588    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 21:41:43.768597 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.768597    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 21:41:43.768654 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.768605    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 21:41:43.768654 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.768611    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 21:41:43.768654 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.768620    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 21:41:43.768654 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.768628    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 21:41:43.769255 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.769237    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 21:41:43.769255 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.769251    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 21:41:43.773040 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.773027    2580 server.go:1263] "Started kubelet"
Oct 30 21:41:43.773145 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.773087    2580 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 21:41:43.773313 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.773248    2580 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 21:41:43.773405 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.773331    2580 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 21:41:43.773817 ip-10-0-130-60 systemd[1]: Started Kubernetes Kubelet.
Oct 30 21:41:43.774846 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.774543    2580 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 21:41:43.774916 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.774850    2580 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 21:41:43.779181 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:41:43.779162    2580 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 30 21:41:43.780337 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.780315    2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 21:41:43.780916 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.780898    2580 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 21:41:43.781063 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:41:43.781024    2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-60.us-west-2.compute.internal\" not found"
Oct 30 21:41:43.781600 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.781584    2580 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 21:41:43.781600 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.781603    2580 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 21:41:43.781756 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.781584    2580 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 21:41:43.784695 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.784662    2580 factory.go:55] Registering systemd factory
Oct 30 21:41:43.784695 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.784697    2580 factory.go:223] Registration of the systemd container factory successfully
Oct 30 21:41:43.784820 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.784666    2580 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 21:41:43.784820 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.784763    2580 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 21:41:43.784954 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.784939    2580 factory.go:153] Registering CRI-O factory
Oct 30 21:41:43.784954 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.784954    2580 factory.go:223] Registration of the crio container factory successfully
Oct 30 21:41:43.785069 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.785009    2580 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 21:41:43.785069 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.785023    2580 factory.go:103] Registering Raw factory
Oct 30 21:41:43.785069 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.785033    2580 manager.go:1199] Started watching for new ooms in manager
Oct 30 21:41:43.785947 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.785935    2580 manager.go:319] Starting recovery of all containers
Oct 30 21:41:43.799086 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.798863    2580 manager.go:324] Recovery completed
Oct 30 21:41:43.803565 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.803549    2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:41:43.805245 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.805220    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-60.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:41:43.805307 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.805256    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-60.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:41:43.805307 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.805267    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-60.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:41:43.805836 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.805819    2580 cpu_manager.go:226] "Starting" policy="none"
Oct 30 21:41:43.805836 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.805833    2580 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 21:41:43.805923 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.805853    2580 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 21:41:43.805970 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.805825    2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:41:43.806628 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.806613    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-60.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:41:43.806710 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.806643    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-60.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:41:43.806710 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.806658    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-60.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:41:43.808002 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.807989    2580 policy_none.go:50] "Start"
Oct 30 21:41:43.808002 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.808004    2580 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 21:41:43.808079 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.808017    2580 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 21:41:43.810029 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.810017    2580 policy_none.go:44] "Start"
Oct 30 21:41:43.839741 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.839694    2580 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 21:41:43.863815 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.844543    2580 manager.go:341] "Starting Device Plugin manager"
Oct 30 21:41:43.863815 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:41:43.844567    2580 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 21:41:43.863815 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.844575    2580 server.go:87] "Starting device plugin registration server"
Oct 30 21:41:43.863815 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.844834    2580 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 21:41:43.863815 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.844847    2580 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 21:41:43.863815 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.844947    2580 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 21:41:43.863815 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.845024    2580 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 21:41:43.863815 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.845030    2580 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 21:41:43.863815 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:41:43.845593    2580 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 21:41:43.863815 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:41:43.845636    2580 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-60.us-west-2.compute.internal\" not found"
Oct 30 21:41:43.863815 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.855156    2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 21:41:43.863815 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.856299    2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 21:41:43.863815 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.856316    2580 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 21:41:43.863815 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.856338    2580 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 21:41:43.863815 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:41:43.856375    2580 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Oct 30 21:41:43.908594 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:41:43.908566    2580 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-130-60.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 30 21:41:43.909497 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:41:43.908567    2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-60.us-west-2.compute.internal.18e36c9e4cd1357b  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-60.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-60.us-west-2.compute.internal,},FirstTimestamp:2026-10-30 21:41:43.773001083 +0000 UTC m=+0.337176319,LastTimestamp:2026-10-30 21:41:43.773001083 +0000 UTC m=+0.337176319,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-60.us-west-2.compute.internal,}"
Oct 30 21:41:43.945081 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.945058    2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:41:43.945945 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.945930    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-60.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:41:43.946024 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.945957    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-60.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:41:43.946024 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.945968    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-60.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:41:43.946024 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.945991    2580 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-130-60.us-west-2.compute.internal"
Oct 30 21:41:43.956755 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.956708    2580 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-60.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-60.us-west-2.compute.internal"]
Oct 30 21:41:43.956809 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.956761    2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:41:43.957486 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.957473    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-60.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:41:43.957543 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.957499    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-60.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:41:43.957543 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.957509    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-60.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:41:43.959669 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.959656    2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:41:43.959846 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.959826    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-60.us-west-2.compute.internal"
Oct 30 21:41:43.959926 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.959874    2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:41:43.960993 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.960807    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-60.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:41:43.961083 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.961012    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-60.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:41:43.961083 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.961055    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-60.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:41:43.961083 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.960883    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-60.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:41:43.961234 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.961092    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-60.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:41:43.961234 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.961103    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-60.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:41:43.963238 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.963222    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-60.us-west-2.compute.internal"
Oct 30 21:41:43.963326 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.963253    2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:41:43.964030 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.964013    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-60.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:41:43.964116 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.964040    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-60.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:41:43.964116 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.964054    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-60.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:41:43.985674 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.985653    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c0b65115e7be9fc3f67a9c81d1477e73-config\") pod \"kube-apiserver-proxy-ip-10-0-130-60.us-west-2.compute.internal\" (UID: \"c0b65115e7be9fc3f67a9c81d1477e73\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-60.us-west-2.compute.internal"
Oct 30 21:41:43.985778 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.985696    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cf62eb3c9c04acf0f549e44a10220f5a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-60.us-west-2.compute.internal\" (UID: \"cf62eb3c9c04acf0f549e44a10220f5a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-60.us-west-2.compute.internal"
Oct 30 21:41:43.985778 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:43.985752    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cf62eb3c9c04acf0f549e44a10220f5a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-60.us-west-2.compute.internal\" (UID: \"cf62eb3c9c04acf0f549e44a10220f5a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-60.us-west-2.compute.internal"
Oct 30 21:41:43.994040 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:41:43.994023    2580 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-60.us-west-2.compute.internal\" not found" node="ip-10-0-130-60.us-west-2.compute.internal"
Oct 30 21:41:43.996585 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:41:43.996570    2580 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-60.us-west-2.compute.internal\" not found" node="ip-10-0-130-60.us-west-2.compute.internal"
Oct 30 21:41:44.002217 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:44.002201    2580 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 21:41:44.002278 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:44.002259    2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-xshzs"
Oct 30 21:41:44.009410 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:44.009393    2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-60.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 30 21:41:44.009492 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:41:44.009470    2580 reflector.go:204] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass"
Oct 30 21:41:44.009579 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:41:44.009562    2580 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Oct 30 21:41:44.009629 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:41:44.009614    2580 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Oct 30 21:41:44.056538 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:41:44.056459    2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-60.us-west-2.compute.internal.18e36c9e4ebd3105  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-60.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-130-60.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-130-60.us-west-2.compute.internal,},FirstTimestamp:2026-10-30 21:41:43.805243653 +0000 UTC m=+0.369418889,LastTimestamp:2026-10-30 21:41:43.805243653 +0000 UTC m=+0.369418889,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-60.us-west-2.compute.internal,}"
Oct 30 21:41:44.062938 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:41:44.062918    2580 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-130-60.us-west-2.compute.internal\" not found" node="ip-10-0-130-60.us-west-2.compute.internal"
Oct 30 21:41:44.063741 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:44.063728    2580 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:41:44.086842 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:44.086814    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c0b65115e7be9fc3f67a9c81d1477e73-config\") pod \"kube-apiserver-proxy-ip-10-0-130-60.us-west-2.compute.internal\" (UID: \"c0b65115e7be9fc3f67a9c81d1477e73\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-60.us-west-2.compute.internal"
Oct 30 21:41:44.086940 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:44.086844    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cf62eb3c9c04acf0f549e44a10220f5a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-60.us-west-2.compute.internal\" (UID: \"cf62eb3c9c04acf0f549e44a10220f5a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-60.us-west-2.compute.internal"
Oct 30 21:41:44.086940 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:44.086869    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cf62eb3c9c04acf0f549e44a10220f5a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-60.us-west-2.compute.internal\" (UID: \"cf62eb3c9c04acf0f549e44a10220f5a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-60.us-west-2.compute.internal"
Oct 30 21:41:44.086940 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:44.086907    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cf62eb3c9c04acf0f549e44a10220f5a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-60.us-west-2.compute.internal\" (UID: \"cf62eb3c9c04acf0f549e44a10220f5a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-60.us-west-2.compute.internal"
Oct 30 21:41:44.086940 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:44.086908    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c0b65115e7be9fc3f67a9c81d1477e73-config\") pod \"kube-apiserver-proxy-ip-10-0-130-60.us-west-2.compute.internal\" (UID: \"c0b65115e7be9fc3f67a9c81d1477e73\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-60.us-west-2.compute.internal"
Oct 30 21:41:44.086940 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:44.086915    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cf62eb3c9c04acf0f549e44a10220f5a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-60.us-west-2.compute.internal\" (UID: \"cf62eb3c9c04acf0f549e44a10220f5a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-60.us-west-2.compute.internal"
Oct 30 21:41:44.208082 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:41:44.208007    2580 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-130-60.us-west-2.compute.internal\" not found" node="ip-10-0-130-60.us-west-2.compute.internal"
Oct 30 21:41:44.263446 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:44.263422    2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:41:44.264316 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:44.264300    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-60.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:41:44.264398 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:44.264330    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-60.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:41:44.264398 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:44.264341    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-60.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:41:44.264398 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:44.264370    2580 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-130-60.us-west-2.compute.internal"
Oct 30 21:41:44.298055 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:44.298033    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-60.us-west-2.compute.internal"
Oct 30 21:41:44.301287 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:44.301269    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-60.us-west-2.compute.internal"
Oct 30 21:41:44.321017 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:44.321000    2580 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-130-60.us-west-2.compute.internal"
Oct 30 21:41:44.381913 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:44.381891    2580 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-60.us-west-2.compute.internal"
Oct 30 21:41:44.538702 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:44.538622    2580 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 21:41:44.539652 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:44.539639    2580 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-60.us-west-2.compute.internal"
Oct 30 21:41:44.596431 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:44.596407    2580 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 21:41:46.478611 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.478588    2580 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 30 21:41:46.479074 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:41:46.478776    2580 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a004d3cd47af64f05b1402e813782859-9b2f76aefe485d1f.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-130-60.us-west-2.compute.internal.18e36c9e4ebd765d\": read tcp 10.0.130.60:39318->3.23.194.188:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-130-60.us-west-2.compute.internal.18e36c9e4ebd765d  default   4796 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-60.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-130-60.us-west-2.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-130-60.us-west-2.compute.internal,},FirstTimestamp:2026-10-30 21:41:43 +0000 UTC,LastTimestamp:2026-10-30 21:41:43.957503589 +0000 UTC m=+0.521678827,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-60.us-west-2.compute.internal,}"
Oct 30 21:41:46.512457 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.512436    2580 apiserver.go:52] "Watching apiserver"
Oct 30 21:41:46.535061 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.535044    2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 30 21:41:46.761879 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.761782    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 21:36:45 +0000 UTC" deadline="2028-07-08 13:08:26.422775036 +0000 UTC"
Oct 30 21:41:46.761879 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.761823    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14799h26m39.660955368s"
Oct 30 21:41:46.777362 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.777346    2580 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 21:41:46.777667 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.777650    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-979r9","openshift-network-operator/iptables-alerter-wbq4l","openshift-ovn-kubernetes/ovnkube-node-22ldv","openshift-image-registry/node-ca-x8gnj","openshift-multus/multus-4nqkj","openshift-multus/multus-additional-cni-plugins-b47v2","openshift-multus/network-metrics-daemon-q6tjc","kube-system/global-pull-secret-syncer-6wbw4","kube-system/kube-apiserver-proxy-ip-10-0-130-60.us-west-2.compute.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qk9dg","openshift-cluster-node-tuning-operator/tuned-hskxk","openshift-dns/node-resolver-2wsrr","kube-system/konnectivity-agent-sztmx","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-60.us-west-2.compute.internal"]
Oct 30 21:41:46.780115 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.780095    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-979r9"
Oct 30 21:41:46.780204 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:41:46.780187    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-979r9" podUID="3784e680-5efe-4ffb-ad71-956d05b57dce"
Oct 30 21:41:46.782072 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.782055    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wbq4l"
Oct 30 21:41:46.784302 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.784284    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-22ldv"
Oct 30 21:41:46.786384 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.786369    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-x8gnj"
Oct 30 21:41:46.788459 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.788444    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4nqkj"
Oct 30 21:41:46.790662 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.790644    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-b47v2"
Oct 30 21:41:46.792777 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.792763    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q6tjc"
Oct 30 21:41:46.792861 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:41:46.792835    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q6tjc" podUID="2205e8e6-f038-47fa-b393-0abe4d622e03"
Oct 30 21:41:46.794875 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.794862    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6wbw4"
Oct 30 21:41:46.794943 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:41:46.794926    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6wbw4" podUID="fe41c70b-7e2c-432f-8389-0282d6fdb332"
Oct 30 21:41:46.797090 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.797074    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qk9dg"
Oct 30 21:41:46.799202 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.799189    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-hskxk"
Oct 30 21:41:46.801358 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.801344    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-2wsrr"
Oct 30 21:41:46.803571 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.803554    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-sztmx"
Oct 30 21:41:46.831713 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.831692    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 21:41:46.831812 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.831750    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 21:41:46.831812 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.831798    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-rx46t\""
Oct 30 21:41:46.831917 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.831857    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 21:41:46.833349 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.833335    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-s2ds7\""
Oct 30 21:41:46.833607 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.833589    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 21:41:46.833607 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.833588    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 21:41:46.833772 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.833592    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 21:41:46.833772 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.833616    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 21:41:46.833772 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.833638    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 21:41:46.833974 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.833959    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 21:41:46.835398 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.835382    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-wvvdp\""
Oct 30 21:41:46.835671 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.835652    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 21:41:46.835671 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.835653    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 21:41:46.835813 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.835694    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 21:41:46.836311 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.836297    2580 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 21:41:46.837763 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.837747    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 21:41:46.837843 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.837827    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-hdwjf\""
Oct 30 21:41:46.837901 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.837877    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 21:41:46.847174 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.847155    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1f3b6a23-d7ea-4ce4-b1e8-68133c4b4ebc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-b47v2\" (UID: \"1f3b6a23-d7ea-4ce4-b1e8-68133c4b4ebc\") " pod="openshift-multus/multus-additional-cni-plugins-b47v2"
Oct 30 21:41:46.847268 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.847181    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/538d5103-1c92-48f1-acc7-f90956848727-run-systemd\") pod \"ovnkube-node-22ldv\" (UID: \"538d5103-1c92-48f1-acc7-f90956848727\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ldv"
Oct 30 21:41:46.847268 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.847197    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/538d5103-1c92-48f1-acc7-f90956848727-log-socket\") pod \"ovnkube-node-22ldv\" (UID: \"538d5103-1c92-48f1-acc7-f90956848727\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ldv"
Oct 30 21:41:46.847268 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.847211    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/538d5103-1c92-48f1-acc7-f90956848727-host-cni-netd\") pod \"ovnkube-node-22ldv\" (UID: \"538d5103-1c92-48f1-acc7-f90956848727\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ldv"
Oct 30 21:41:46.847268 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.847229    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a2e3b25e-b6dc-46ef-86e8-4e70f9dad200-hostroot\") pod \"multus-4nqkj\" (UID: \"a2e3b25e-b6dc-46ef-86e8-4e70f9dad200\") " pod="openshift-multus/multus-4nqkj"
Oct 30 21:41:46.847468 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.847292    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ed08bc51-70e6-4658-adb3-ed36f5176552-host\") pod \"tuned-hskxk\" (UID: \"ed08bc51-70e6-4658-adb3-ed36f5176552\") " pod="openshift-cluster-node-tuning-operator/tuned-hskxk"
Oct 30 21:41:46.847468 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.847332    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b227ad6a-9e59-415e-b7e6-f7c25e988abb-host\") pod \"node-ca-x8gnj\" (UID: \"b227ad6a-9e59-415e-b7e6-f7c25e988abb\") " pod="openshift-image-registry/node-ca-x8gnj"
Oct 30 21:41:46.847468 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.847360    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0929b9f6-12ec-4f4d-8ba3-5cef9defba15-iptables-alerter-script\") pod \"iptables-alerter-wbq4l\" (UID: \"0929b9f6-12ec-4f4d-8ba3-5cef9defba15\") " pod="openshift-network-operator/iptables-alerter-wbq4l"
Oct 30 21:41:46.847468 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.847385    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/538d5103-1c92-48f1-acc7-f90956848727-host-slash\") pod \"ovnkube-node-22ldv\" (UID: \"538d5103-1c92-48f1-acc7-f90956848727\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ldv"
Oct 30 21:41:46.847468 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.847410    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8b7v\" (UniqueName: \"kubernetes.io/projected/a2e3b25e-b6dc-46ef-86e8-4e70f9dad200-kube-api-access-p8b7v\") pod \"multus-4nqkj\" (UID: \"a2e3b25e-b6dc-46ef-86e8-4e70f9dad200\") " pod="openshift-multus/multus-4nqkj"
Oct 30 21:41:46.847689 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.847468    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4427f59d-8003-440f-9646-a46b176cc4a7-device-dir\") pod \"aws-ebs-csi-driver-node-qk9dg\" (UID: \"4427f59d-8003-440f-9646-a46b176cc4a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qk9dg"
Oct 30 21:41:46.847689 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.847506    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ed08bc51-70e6-4658-adb3-ed36f5176552-etc-tuned\") pod \"tuned-hskxk\" (UID: \"ed08bc51-70e6-4658-adb3-ed36f5176552\") " pod="openshift-cluster-node-tuning-operator/tuned-hskxk"
Oct 30 21:41:46.847689 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.847580    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fe41c70b-7e2c-432f-8389-0282d6fdb332-dbus\") pod \"global-pull-secret-syncer-6wbw4\" (UID: \"fe41c70b-7e2c-432f-8389-0282d6fdb332\") " pod="kube-system/global-pull-secret-syncer-6wbw4"
Oct 30 21:41:46.847689 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.847618    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fe41c70b-7e2c-432f-8389-0282d6fdb332-original-pull-secret\") pod \"global-pull-secret-syncer-6wbw4\" (UID: \"fe41c70b-7e2c-432f-8389-0282d6fdb332\") " pod="kube-system/global-pull-secret-syncer-6wbw4"
Oct 30 21:41:46.847689 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.847651    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/538d5103-1c92-48f1-acc7-f90956848727-etc-openvswitch\") pod \"ovnkube-node-22ldv\" (UID: \"538d5103-1c92-48f1-acc7-f90956848727\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ldv"
Oct 30 21:41:46.847689 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.847677    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/538d5103-1c92-48f1-acc7-f90956848727-node-log\") pod \"ovnkube-node-22ldv\" (UID: \"538d5103-1c92-48f1-acc7-f90956848727\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ldv"
Oct 30 21:41:46.847948 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.847704    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1f3b6a23-d7ea-4ce4-b1e8-68133c4b4ebc-cnibin\") pod \"multus-additional-cni-plugins-b47v2\" (UID: \"1f3b6a23-d7ea-4ce4-b1e8-68133c4b4ebc\") " pod="openshift-multus/multus-additional-cni-plugins-b47v2"
Oct 30 21:41:46.847948 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.847732    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrpz5\" (UniqueName: \"kubernetes.io/projected/538d5103-1c92-48f1-acc7-f90956848727-kube-api-access-hrpz5\") pod \"ovnkube-node-22ldv\" (UID: \"538d5103-1c92-48f1-acc7-f90956848727\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ldv"
Oct 30 21:41:46.847948 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.847754    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a2e3b25e-b6dc-46ef-86e8-4e70f9dad200-system-cni-dir\") pod \"multus-4nqkj\" (UID: \"a2e3b25e-b6dc-46ef-86e8-4e70f9dad200\") " pod="openshift-multus/multus-4nqkj"
Oct 30 21:41:46.847948 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.847790    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ed08bc51-70e6-4658-adb3-ed36f5176552-lib-modules\") pod \"tuned-hskxk\" (UID: \"ed08bc51-70e6-4658-adb3-ed36f5176552\") " pod="openshift-cluster-node-tuning-operator/tuned-hskxk"
Oct 30 21:41:46.847948 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.847812    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hkfh\" (UniqueName: \"kubernetes.io/projected/71ddf455-cbf7-49a6-9773-30f7e4997a39-kube-api-access-7hkfh\") pod \"node-resolver-2wsrr\" (UID: \"71ddf455-cbf7-49a6-9773-30f7e4997a39\") " pod="openshift-dns/node-resolver-2wsrr"
Oct 30 21:41:46.847948 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.847827    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/538d5103-1c92-48f1-acc7-f90956848727-systemd-units\") pod \"ovnkube-node-22ldv\" (UID: \"538d5103-1c92-48f1-acc7-f90956848727\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ldv"
Oct 30 21:41:46.847948 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.847840    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/538d5103-1c92-48f1-acc7-f90956848727-run-ovn\") pod \"ovnkube-node-22ldv\" (UID: \"538d5103-1c92-48f1-acc7-f90956848727\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ldv"
Oct 30 21:41:46.847948 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.847863    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/538d5103-1c92-48f1-acc7-f90956848727-host-cni-bin\") pod \"ovnkube-node-22ldv\" (UID: \"538d5103-1c92-48f1-acc7-f90956848727\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ldv"
Oct 30 21:41:46.847948 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.847894    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a2e3b25e-b6dc-46ef-86e8-4e70f9dad200-cni-binary-copy\") pod \"multus-4nqkj\" (UID: \"a2e3b25e-b6dc-46ef-86e8-4e70f9dad200\") " pod="openshift-multus/multus-4nqkj"
Oct 30 21:41:46.847948 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.847922    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nc76p\" (UniqueName: \"kubernetes.io/projected/4427f59d-8003-440f-9646-a46b176cc4a7-kube-api-access-nc76p\") pod \"aws-ebs-csi-driver-node-qk9dg\" (UID: \"4427f59d-8003-440f-9646-a46b176cc4a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qk9dg"
Oct 30 21:41:46.847948 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.847947    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ed08bc51-70e6-4658-adb3-ed36f5176552-etc-sysctl-conf\") pod \"tuned-hskxk\" (UID: \"ed08bc51-70e6-4658-adb3-ed36f5176552\") " pod="openshift-cluster-node-tuning-operator/tuned-hskxk"
Oct 30 21:41:46.848290 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.847968    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/538d5103-1c92-48f1-acc7-f90956848727-run-openvswitch\") pod \"ovnkube-node-22ldv\" (UID: \"538d5103-1c92-48f1-acc7-f90956848727\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ldv"
Oct 30 21:41:46.848290 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.847988    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a2e3b25e-b6dc-46ef-86e8-4e70f9dad200-host-run-k8s-cni-cncf-io\") pod \"multus-4nqkj\" (UID: \"a2e3b25e-b6dc-46ef-86e8-4e70f9dad200\") " pod="openshift-multus/multus-4nqkj"
Oct 30 21:41:46.848290 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.848009    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a2e3b25e-b6dc-46ef-86e8-4e70f9dad200-multus-socket-dir-parent\") pod \"multus-4nqkj\" (UID: \"a2e3b25e-b6dc-46ef-86e8-4e70f9dad200\") " pod="openshift-multus/multus-4nqkj"
Oct 30 21:41:46.848290 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.848040    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a2e3b25e-b6dc-46ef-86e8-4e70f9dad200-host-var-lib-cni-multus\") pod \"multus-4nqkj\" (UID: \"a2e3b25e-b6dc-46ef-86e8-4e70f9dad200\") " pod="openshift-multus/multus-4nqkj"
Oct 30 21:41:46.848290 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.848057    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a2e3b25e-b6dc-46ef-86e8-4e70f9dad200-multus-conf-dir\") pod \"multus-4nqkj\" (UID: \"a2e3b25e-b6dc-46ef-86e8-4e70f9dad200\") " pod="openshift-multus/multus-4nqkj"
Oct 30 21:41:46.848290 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.848075    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ed08bc51-70e6-4658-adb3-ed36f5176552-etc-sysconfig\") pod \"tuned-hskxk\" (UID: \"ed08bc51-70e6-4658-adb3-ed36f5176552\") " pod="openshift-cluster-node-tuning-operator/tuned-hskxk"
Oct 30 21:41:46.848290 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.848108    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ed08bc51-70e6-4658-adb3-ed36f5176552-etc-kubernetes\") pod \"tuned-hskxk\" (UID: \"ed08bc51-70e6-4658-adb3-ed36f5176552\") " pod="openshift-cluster-node-tuning-operator/tuned-hskxk"
Oct 30 21:41:46.848290 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.848155    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ed08bc51-70e6-4658-adb3-ed36f5176552-tmp\") pod \"tuned-hskxk\" (UID: \"ed08bc51-70e6-4658-adb3-ed36f5176552\") " pod="openshift-cluster-node-tuning-operator/tuned-hskxk"
Oct 30 21:41:46.848290 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.848190    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4427f59d-8003-440f-9646-a46b176cc4a7-socket-dir\") pod \"aws-ebs-csi-driver-node-qk9dg\" (UID: \"4427f59d-8003-440f-9646-a46b176cc4a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qk9dg"
Oct 30 21:41:46.848290 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.848214    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ed08bc51-70e6-4658-adb3-ed36f5176552-etc-modprobe-d\") pod \"tuned-hskxk\" (UID: \"ed08bc51-70e6-4658-adb3-ed36f5176552\") " pod="openshift-cluster-node-tuning-operator/tuned-hskxk"
Oct 30 21:41:46.848290 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.848228    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ed08bc51-70e6-4658-adb3-ed36f5176552-etc-sysctl-d\") pod \"tuned-hskxk\" (UID: \"ed08bc51-70e6-4658-adb3-ed36f5176552\") " pod="openshift-cluster-node-tuning-operator/tuned-hskxk"
Oct 30 21:41:46.848290 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.848268    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/238ff7d8-7da8-4cd8-9d45-deedf615211e-konnectivity-ca\") pod \"konnectivity-agent-sztmx\" (UID: \"238ff7d8-7da8-4cd8-9d45-deedf615211e\") " pod="kube-system/konnectivity-agent-sztmx"
Oct 30 21:41:46.848706 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.848317    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpbtv\" (UniqueName: \"kubernetes.io/projected/b227ad6a-9e59-415e-b7e6-f7c25e988abb-kube-api-access-qpbtv\") pod \"node-ca-x8gnj\" (UID: \"b227ad6a-9e59-415e-b7e6-f7c25e988abb\") " pod="openshift-image-registry/node-ca-x8gnj"
Oct 30 21:41:46.848706 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.848362    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0929b9f6-12ec-4f4d-8ba3-5cef9defba15-host-slash\") pod \"iptables-alerter-wbq4l\" (UID: \"0929b9f6-12ec-4f4d-8ba3-5cef9defba15\") " pod="openshift-network-operator/iptables-alerter-wbq4l"
Oct 30 21:41:46.848706 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.848388    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/538d5103-1c92-48f1-acc7-f90956848727-env-overrides\") pod \"ovnkube-node-22ldv\" (UID: \"538d5103-1c92-48f1-acc7-f90956848727\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ldv"
Oct 30 21:41:46.848706 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.848412    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a2e3b25e-b6dc-46ef-86e8-4e70f9dad200-multus-daemon-config\") pod \"multus-4nqkj\" (UID: \"a2e3b25e-b6dc-46ef-86e8-4e70f9dad200\") " pod="openshift-multus/multus-4nqkj"
Oct 30 21:41:46.848706 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.848439    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1f3b6a23-d7ea-4ce4-b1e8-68133c4b4ebc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-b47v2\" (UID: \"1f3b6a23-d7ea-4ce4-b1e8-68133c4b4ebc\") " pod="openshift-multus/multus-additional-cni-plugins-b47v2"
Oct 30 21:41:46.848706 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.848461    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4427f59d-8003-440f-9646-a46b176cc4a7-registration-dir\") pod \"aws-ebs-csi-driver-node-qk9dg\" (UID: \"4427f59d-8003-440f-9646-a46b176cc4a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qk9dg"
Oct 30 21:41:46.848706 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.848482    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pd6bv\" (UniqueName: \"kubernetes.io/projected/ed08bc51-70e6-4658-adb3-ed36f5176552-kube-api-access-pd6bv\") pod \"tuned-hskxk\" (UID: \"ed08bc51-70e6-4658-adb3-ed36f5176552\") " pod="openshift-cluster-node-tuning-operator/tuned-hskxk"
Oct 30 21:41:46.848706 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.848510    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/538d5103-1c92-48f1-acc7-f90956848727-host-kubelet\") pod \"ovnkube-node-22ldv\" (UID: \"538d5103-1c92-48f1-acc7-f90956848727\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ldv"
Oct 30 21:41:46.848706 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.848569    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/538d5103-1c92-48f1-acc7-f90956848727-host-run-ovn-kubernetes\") pod \"ovnkube-node-22ldv\" (UID: \"538d5103-1c92-48f1-acc7-f90956848727\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ldv"
Oct 30 21:41:46.848706 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.848589    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1f3b6a23-d7ea-4ce4-b1e8-68133c4b4ebc-cni-binary-copy\") pod \"multus-additional-cni-plugins-b47v2\" (UID: \"1f3b6a23-d7ea-4ce4-b1e8-68133c4b4ebc\") " pod="openshift-multus/multus-additional-cni-plugins-b47v2"
Oct 30 21:41:46.848706 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.848630    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ed08bc51-70e6-4658-adb3-ed36f5176552-sys\") pod \"tuned-hskxk\" (UID: \"ed08bc51-70e6-4658-adb3-ed36f5176552\") " pod="openshift-cluster-node-tuning-operator/tuned-hskxk"
Oct 30 21:41:46.848706 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.848652    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ed08bc51-70e6-4658-adb3-ed36f5176552-var-lib-kubelet\") pod \"tuned-hskxk\" (UID: \"ed08bc51-70e6-4658-adb3-ed36f5176552\") " pod="openshift-cluster-node-tuning-operator/tuned-hskxk"
Oct 30 21:41:46.848706 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.848666    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/238ff7d8-7da8-4cd8-9d45-deedf615211e-agent-certs\") pod \"konnectivity-agent-sztmx\" (UID: \"238ff7d8-7da8-4cd8-9d45-deedf615211e\") " pod="kube-system/konnectivity-agent-sztmx"
Oct 30 21:41:46.848706 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.848692    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-579v9\" (UniqueName: \"kubernetes.io/projected/3784e680-5efe-4ffb-ad71-956d05b57dce-kube-api-access-579v9\") pod \"network-check-target-979r9\" (UID: \"3784e680-5efe-4ffb-ad71-956d05b57dce\") " pod="openshift-network-diagnostics/network-check-target-979r9"
Oct 30 21:41:46.849236 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.848728    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1f3b6a23-d7ea-4ce4-b1e8-68133c4b4ebc-system-cni-dir\") pod \"multus-additional-cni-plugins-b47v2\" (UID: \"1f3b6a23-d7ea-4ce4-b1e8-68133c4b4ebc\") " pod="openshift-multus/multus-additional-cni-plugins-b47v2"
Oct 30 21:41:46.849236 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.848766    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/538d5103-1c92-48f1-acc7-f90956848727-ovnkube-script-lib\") pod \"ovnkube-node-22ldv\" (UID: \"538d5103-1c92-48f1-acc7-f90956848727\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ldv"
Oct 30 21:41:46.849236 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.848809    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4427f59d-8003-440f-9646-a46b176cc4a7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qk9dg\" (UID: \"4427f59d-8003-440f-9646-a46b176cc4a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qk9dg"
Oct 30 21:41:46.849236 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.848835    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4427f59d-8003-440f-9646-a46b176cc4a7-etc-selinux\") pod \"aws-ebs-csi-driver-node-qk9dg\" (UID: \"4427f59d-8003-440f-9646-a46b176cc4a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qk9dg"
Oct 30 21:41:46.849236 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.848857    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a2e3b25e-b6dc-46ef-86e8-4e70f9dad200-etc-kubernetes\") pod \"multus-4nqkj\" (UID: \"a2e3b25e-b6dc-46ef-86e8-4e70f9dad200\") " pod="openshift-multus/multus-4nqkj"
Oct 30 21:41:46.849236 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.848871    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b227ad6a-9e59-415e-b7e6-f7c25e988abb-serviceca\") pod \"node-ca-x8gnj\" (UID: \"b227ad6a-9e59-415e-b7e6-f7c25e988abb\") " pod="openshift-image-registry/node-ca-x8gnj"
Oct 30 21:41:46.849236 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.848890    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fe41c70b-7e2c-432f-8389-0282d6fdb332-kubelet-config\") pod \"global-pull-secret-syncer-6wbw4\" (UID: \"fe41c70b-7e2c-432f-8389-0282d6fdb332\") " pod="kube-system/global-pull-secret-syncer-6wbw4"
Oct 30 21:41:46.849236 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.848909    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/538d5103-1c92-48f1-acc7-f90956848727-host-run-netns\") pod \"ovnkube-node-22ldv\" (UID: \"538d5103-1c92-48f1-acc7-f90956848727\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ldv"
Oct 30 21:41:46.849236 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.848922    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a2e3b25e-b6dc-46ef-86e8-4e70f9dad200-multus-cni-dir\") pod \"multus-4nqkj\" (UID: \"a2e3b25e-b6dc-46ef-86e8-4e70f9dad200\") " pod="openshift-multus/multus-4nqkj"
Oct 30 21:41:46.849236 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.848945    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a2e3b25e-b6dc-46ef-86e8-4e70f9dad200-cnibin\") pod \"multus-4nqkj\" (UID: \"a2e3b25e-b6dc-46ef-86e8-4e70f9dad200\") " pod="openshift-multus/multus-4nqkj"
Oct 30 21:41:46.849236 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.848980    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a2e3b25e-b6dc-46ef-86e8-4e70f9dad200-host-var-lib-cni-bin\") pod \"multus-4nqkj\" (UID: \"a2e3b25e-b6dc-46ef-86e8-4e70f9dad200\") " pod="openshift-multus/multus-4nqkj"
Oct 30 21:41:46.849236 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.849006    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gq9m\" (UniqueName: \"kubernetes.io/projected/1f3b6a23-d7ea-4ce4-b1e8-68133c4b4ebc-kube-api-access-7gq9m\") pod \"multus-additional-cni-plugins-b47v2\" (UID: \"1f3b6a23-d7ea-4ce4-b1e8-68133c4b4ebc\") " pod="openshift-multus/multus-additional-cni-plugins-b47v2"
Oct 30 21:41:46.849236 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.849086    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2205e8e6-f038-47fa-b393-0abe4d622e03-metrics-certs\") pod \"network-metrics-daemon-q6tjc\" (UID: \"2205e8e6-f038-47fa-b393-0abe4d622e03\") " pod="openshift-multus/network-metrics-daemon-q6tjc"
Oct 30 21:41:46.849236 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.849115    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swjcw\" (UniqueName: \"kubernetes.io/projected/0929b9f6-12ec-4f4d-8ba3-5cef9defba15-kube-api-access-swjcw\") pod \"iptables-alerter-wbq4l\" (UID: \"0929b9f6-12ec-4f4d-8ba3-5cef9defba15\") " pod="openshift-network-operator/iptables-alerter-wbq4l"
Oct 30 21:41:46.849236 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.849147    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/538d5103-1c92-48f1-acc7-f90956848727-ovnkube-config\") pod \"ovnkube-node-22ldv\" (UID: \"538d5103-1c92-48f1-acc7-f90956848727\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ldv"
Oct 30 21:41:46.849236 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.849162    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a2e3b25e-b6dc-46ef-86e8-4e70f9dad200-host-var-lib-kubelet\") pod \"multus-4nqkj\" (UID: \"a2e3b25e-b6dc-46ef-86e8-4e70f9dad200\") " pod="openshift-multus/multus-4nqkj"
Oct 30 21:41:46.849759 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.849189    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ed08bc51-70e6-4658-adb3-ed36f5176552-etc-systemd\") pod \"tuned-hskxk\" (UID: \"ed08bc51-70e6-4658-adb3-ed36f5176552\") " pod="openshift-cluster-node-tuning-operator/tuned-hskxk"
Oct 30 21:41:46.849759 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.849219    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/71ddf455-cbf7-49a6-9773-30f7e4997a39-hosts-file\") pod \"node-resolver-2wsrr\" (UID: \"71ddf455-cbf7-49a6-9773-30f7e4997a39\") " pod="openshift-dns/node-resolver-2wsrr"
Oct 30 21:41:46.849759 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.849238    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/538d5103-1c92-48f1-acc7-f90956848727-ovn-node-metrics-cert\") pod \"ovnkube-node-22ldv\" (UID: \"538d5103-1c92-48f1-acc7-f90956848727\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ldv"
Oct 30 21:41:46.849759 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.849264    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a2e3b25e-b6dc-46ef-86e8-4e70f9dad200-host-run-multus-certs\") pod \"multus-4nqkj\" (UID: \"a2e3b25e-b6dc-46ef-86e8-4e70f9dad200\") " pod="openshift-multus/multus-4nqkj"
Oct 30 21:41:46.849759 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.849294    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4427f59d-8003-440f-9646-a46b176cc4a7-sys-fs\") pod \"aws-ebs-csi-driver-node-qk9dg\" (UID: \"4427f59d-8003-440f-9646-a46b176cc4a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qk9dg"
Oct 30 21:41:46.849759 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.849344    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ed08bc51-70e6-4658-adb3-ed36f5176552-run\") pod \"tuned-hskxk\" (UID: \"ed08bc51-70e6-4658-adb3-ed36f5176552\") " pod="openshift-cluster-node-tuning-operator/tuned-hskxk"
Oct 30 21:41:46.849759 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.849372    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbwzr\" (UniqueName: \"kubernetes.io/projected/2205e8e6-f038-47fa-b393-0abe4d622e03-kube-api-access-rbwzr\") pod \"network-metrics-daemon-q6tjc\" (UID: \"2205e8e6-f038-47fa-b393-0abe4d622e03\") " pod="openshift-multus/network-metrics-daemon-q6tjc"
Oct 30 21:41:46.849759 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.849391    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/538d5103-1c92-48f1-acc7-f90956848727-var-lib-openvswitch\") pod \"ovnkube-node-22ldv\" (UID: \"538d5103-1c92-48f1-acc7-f90956848727\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ldv"
Oct 30 21:41:46.849759 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.849412    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/538d5103-1c92-48f1-acc7-f90956848727-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-22ldv\" (UID: \"538d5103-1c92-48f1-acc7-f90956848727\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ldv"
Oct 30 21:41:46.849759 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.849437    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a2e3b25e-b6dc-46ef-86e8-4e70f9dad200-host-run-netns\") pod \"multus-4nqkj\" (UID: \"a2e3b25e-b6dc-46ef-86e8-4e70f9dad200\") " pod="openshift-multus/multus-4nqkj"
Oct 30 21:41:46.849759 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.849450    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1f3b6a23-d7ea-4ce4-b1e8-68133c4b4ebc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-b47v2\" (UID: \"1f3b6a23-d7ea-4ce4-b1e8-68133c4b4ebc\") " pod="openshift-multus/multus-additional-cni-plugins-b47v2"
Oct 30 21:41:46.849759 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.849475    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71ddf455-cbf7-49a6-9773-30f7e4997a39-tmp-dir\") pod \"node-resolver-2wsrr\" (UID: \"71ddf455-cbf7-49a6-9773-30f7e4997a39\") " pod="openshift-dns/node-resolver-2wsrr"
Oct 30 21:41:46.873973 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.873954    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 21:41:46.874072 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.873972    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 21:41:46.874072 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.873974    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 21:41:46.874072 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.874044    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-2qk7p\""
Oct 30 21:41:46.877021 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.877007    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 21:41:46.879333 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.879317    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-58hb8\""
Oct 30 21:41:46.879386 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.879366    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 21:41:46.879432 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.879418    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 21:41:46.879474 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.879441    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 21:41:46.879474 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.879440    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-48hj5\""
Oct 30 21:41:46.880851 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.880837    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 21:41:46.880922 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.880842    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 21:41:46.880922 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.880894    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 21:41:46.881359 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.881345    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-5r6kl\""
Oct 30 21:41:46.881432 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.881420    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 21:41:46.881561 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.881545    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-vwlkq\""
Oct 30 21:41:46.881610 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.881552    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 21:41:46.881655 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.881643    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 21:41:46.921437 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.921421    2580 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:41:46.921522 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.921465    2580 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 21:41:46.921598 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.921543    2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-lpmgv"
Oct 30 21:41:46.950380 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.950358    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/538d5103-1c92-48f1-acc7-f90956848727-systemd-units\") pod \"ovnkube-node-22ldv\" (UID: \"538d5103-1c92-48f1-acc7-f90956848727\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ldv"
Oct 30 21:41:46.950486 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.950388    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/538d5103-1c92-48f1-acc7-f90956848727-run-ovn\") pod \"ovnkube-node-22ldv\" (UID: \"538d5103-1c92-48f1-acc7-f90956848727\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ldv"
Oct 30 21:41:46.950486 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.950413    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/538d5103-1c92-48f1-acc7-f90956848727-host-cni-bin\") pod \"ovnkube-node-22ldv\" (UID: \"538d5103-1c92-48f1-acc7-f90956848727\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ldv"
Oct 30 21:41:46.950486 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.950435    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a2e3b25e-b6dc-46ef-86e8-4e70f9dad200-cni-binary-copy\") pod \"multus-4nqkj\" (UID: \"a2e3b25e-b6dc-46ef-86e8-4e70f9dad200\") " pod="openshift-multus/multus-4nqkj"
Oct 30 21:41:46.950486 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.950445    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/538d5103-1c92-48f1-acc7-f90956848727-systemd-units\") pod \"ovnkube-node-22ldv\" (UID: \"538d5103-1c92-48f1-acc7-f90956848727\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ldv"
Oct 30 21:41:46.950486 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.950457    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nc76p\" (UniqueName: \"kubernetes.io/projected/4427f59d-8003-440f-9646-a46b176cc4a7-kube-api-access-nc76p\") pod \"aws-ebs-csi-driver-node-qk9dg\" (UID: \"4427f59d-8003-440f-9646-a46b176cc4a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qk9dg"
Oct 30 21:41:46.950754 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.950496    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ed08bc51-70e6-4658-adb3-ed36f5176552-etc-sysctl-conf\") pod \"tuned-hskxk\" (UID: \"ed08bc51-70e6-4658-adb3-ed36f5176552\") " pod="openshift-cluster-node-tuning-operator/tuned-hskxk"
Oct 30 21:41:46.950754 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.950501    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/538d5103-1c92-48f1-acc7-f90956848727-host-cni-bin\") pod \"ovnkube-node-22ldv\" (UID: \"538d5103-1c92-48f1-acc7-f90956848727\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ldv"
Oct 30 21:41:46.950754 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.950522    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/538d5103-1c92-48f1-acc7-f90956848727-run-openvswitch\") pod \"ovnkube-node-22ldv\" (UID: \"538d5103-1c92-48f1-acc7-f90956848727\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ldv"
Oct 30 21:41:46.950754 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.950544    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/538d5103-1c92-48f1-acc7-f90956848727-run-ovn\") pod \"ovnkube-node-22ldv\" (UID: \"538d5103-1c92-48f1-acc7-f90956848727\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ldv"
Oct 30 21:41:46.950754 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.950564    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/538d5103-1c92-48f1-acc7-f90956848727-run-openvswitch\") pod \"ovnkube-node-22ldv\" (UID: \"538d5103-1c92-48f1-acc7-f90956848727\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ldv"
Oct 30 21:41:46.950754 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.950570    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a2e3b25e-b6dc-46ef-86e8-4e70f9dad200-host-run-k8s-cni-cncf-io\") pod \"multus-4nqkj\" (UID: \"a2e3b25e-b6dc-46ef-86e8-4e70f9dad200\") " pod="openshift-multus/multus-4nqkj"
Oct 30 21:41:46.950754 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.950605    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a2e3b25e-b6dc-46ef-86e8-4e70f9dad200-multus-socket-dir-parent\") pod \"multus-4nqkj\" (UID: \"a2e3b25e-b6dc-46ef-86e8-4e70f9dad200\") " pod="openshift-multus/multus-4nqkj"
Oct 30 21:41:46.950754 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.950613    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a2e3b25e-b6dc-46ef-86e8-4e70f9dad200-host-run-k8s-cni-cncf-io\") pod \"multus-4nqkj\" (UID: \"a2e3b25e-b6dc-46ef-86e8-4e70f9dad200\") " pod="openshift-multus/multus-4nqkj"
Oct 30 21:41:46.950754 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.950629    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a2e3b25e-b6dc-46ef-86e8-4e70f9dad200-host-var-lib-cni-multus\") pod \"multus-4nqkj\" (UID: \"a2e3b25e-b6dc-46ef-86e8-4e70f9dad200\") " pod="openshift-multus/multus-4nqkj"
Oct 30 21:41:46.950754 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.950650    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a2e3b25e-b6dc-46ef-86e8-4e70f9dad200-multus-conf-dir\") pod \"multus-4nqkj\" (UID: \"a2e3b25e-b6dc-46ef-86e8-4e70f9dad200\") " pod="openshift-multus/multus-4nqkj"
Oct 30 21:41:46.950754 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.950648    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ed08bc51-70e6-4658-adb3-ed36f5176552-etc-sysctl-conf\") pod \"tuned-hskxk\" (UID: \"ed08bc51-70e6-4658-adb3-ed36f5176552\") " pod="openshift-cluster-node-tuning-operator/tuned-hskxk"
Oct 30 21:41:46.950754 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.950669    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ed08bc51-70e6-4658-adb3-ed36f5176552-etc-sysconfig\") pod \"tuned-hskxk\" (UID: \"ed08bc51-70e6-4658-adb3-ed36f5176552\") " pod="openshift-cluster-node-tuning-operator/tuned-hskxk"
Oct 30 21:41:46.950754 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.950690    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ed08bc51-70e6-4658-adb3-ed36f5176552-etc-kubernetes\") pod \"tuned-hskxk\" (UID: \"ed08bc51-70e6-4658-adb3-ed36f5176552\") " pod="openshift-cluster-node-tuning-operator/tuned-hskxk"
Oct 30 21:41:46.950754 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.950708    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ed08bc51-70e6-4658-adb3-ed36f5176552-tmp\") pod \"tuned-hskxk\" (UID: \"ed08bc51-70e6-4658-adb3-ed36f5176552\") " pod="openshift-cluster-node-tuning-operator/tuned-hskxk"
Oct 30 21:41:46.950754 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.950734    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a2e3b25e-b6dc-46ef-86e8-4e70f9dad200-multus-socket-dir-parent\") pod \"multus-4nqkj\" (UID: \"a2e3b25e-b6dc-46ef-86e8-4e70f9dad200\") " pod="openshift-multus/multus-4nqkj"
Oct 30 21:41:46.950754 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.950703    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a2e3b25e-b6dc-46ef-86e8-4e70f9dad200-host-var-lib-cni-multus\") pod \"multus-4nqkj\" (UID: \"a2e3b25e-b6dc-46ef-86e8-4e70f9dad200\") " pod="openshift-multus/multus-4nqkj"
Oct 30 21:41:46.950754 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.950738    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a2e3b25e-b6dc-46ef-86e8-4e70f9dad200-multus-conf-dir\") pod \"multus-4nqkj\" (UID: \"a2e3b25e-b6dc-46ef-86e8-4e70f9dad200\") " pod="openshift-multus/multus-4nqkj"
Oct 30 21:41:46.951592 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.950771    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ed08bc51-70e6-4658-adb3-ed36f5176552-etc-sysconfig\") pod \"tuned-hskxk\" (UID: \"ed08bc51-70e6-4658-adb3-ed36f5176552\") " pod="openshift-cluster-node-tuning-operator/tuned-hskxk"
Oct 30 21:41:46.951592 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.950795    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4427f59d-8003-440f-9646-a46b176cc4a7-socket-dir\") pod \"aws-ebs-csi-driver-node-qk9dg\" (UID: \"4427f59d-8003-440f-9646-a46b176cc4a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qk9dg"
Oct 30 21:41:46.951592 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.950796    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ed08bc51-70e6-4658-adb3-ed36f5176552-etc-kubernetes\") pod \"tuned-hskxk\" (UID: \"ed08bc51-70e6-4658-adb3-ed36f5176552\") " pod="openshift-cluster-node-tuning-operator/tuned-hskxk"
Oct 30 21:41:46.951592 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.950813    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ed08bc51-70e6-4658-adb3-ed36f5176552-etc-modprobe-d\") pod \"tuned-hskxk\" (UID: \"ed08bc51-70e6-4658-adb3-ed36f5176552\") " pod="openshift-cluster-node-tuning-operator/tuned-hskxk"
Oct 30 21:41:46.951592 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.950836    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ed08bc51-70e6-4658-adb3-ed36f5176552-etc-sysctl-d\") pod \"tuned-hskxk\" (UID: \"ed08bc51-70e6-4658-adb3-ed36f5176552\") " pod="openshift-cluster-node-tuning-operator/tuned-hskxk"
Oct 30 21:41:46.951592 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.950851    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/238ff7d8-7da8-4cd8-9d45-deedf615211e-konnectivity-ca\") pod \"konnectivity-agent-sztmx\" (UID: \"238ff7d8-7da8-4cd8-9d45-deedf615211e\") " pod="kube-system/konnectivity-agent-sztmx"
Oct 30 21:41:46.951592 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.950866    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpbtv\" (UniqueName: \"kubernetes.io/projected/b227ad6a-9e59-415e-b7e6-f7c25e988abb-kube-api-access-qpbtv\") pod \"node-ca-x8gnj\" (UID: \"b227ad6a-9e59-415e-b7e6-f7c25e988abb\") " pod="openshift-image-registry/node-ca-x8gnj"
Oct 30 21:41:46.951592 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.950880    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0929b9f6-12ec-4f4d-8ba3-5cef9defba15-host-slash\") pod \"iptables-alerter-wbq4l\" (UID: \"0929b9f6-12ec-4f4d-8ba3-5cef9defba15\") " pod="openshift-network-operator/iptables-alerter-wbq4l"
Oct 30 21:41:46.951592 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.950895    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4427f59d-8003-440f-9646-a46b176cc4a7-socket-dir\") pod \"aws-ebs-csi-driver-node-qk9dg\" (UID: \"4427f59d-8003-440f-9646-a46b176cc4a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qk9dg"
Oct 30 21:41:46.951592 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.950896    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/538d5103-1c92-48f1-acc7-f90956848727-env-overrides\") pod \"ovnkube-node-22ldv\" (UID: \"538d5103-1c92-48f1-acc7-f90956848727\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ldv"
Oct 30 21:41:46.951592 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.950942    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a2e3b25e-b6dc-46ef-86e8-4e70f9dad200-multus-daemon-config\") pod \"multus-4nqkj\" (UID: \"a2e3b25e-b6dc-46ef-86e8-4e70f9dad200\") " pod="openshift-multus/multus-4nqkj"
Oct 30 21:41:46.951592 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.950970    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1f3b6a23-d7ea-4ce4-b1e8-68133c4b4ebc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-b47v2\" (UID: \"1f3b6a23-d7ea-4ce4-b1e8-68133c4b4ebc\") " pod="openshift-multus/multus-additional-cni-plugins-b47v2"
Oct 30 21:41:46.951592 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.950977    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a2e3b25e-b6dc-46ef-86e8-4e70f9dad200-cni-binary-copy\") pod \"multus-4nqkj\" (UID: \"a2e3b25e-b6dc-46ef-86e8-4e70f9dad200\") " pod="openshift-multus/multus-4nqkj"
Oct 30 21:41:46.951592 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.950995    2580 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 21:41:46.951592 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.950995    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4427f59d-8003-440f-9646-a46b176cc4a7-registration-dir\") pod \"aws-ebs-csi-driver-node-qk9dg\" (UID: \"4427f59d-8003-440f-9646-a46b176cc4a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qk9dg"
Oct 30 21:41:46.951592 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.950981    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ed08bc51-70e6-4658-adb3-ed36f5176552-etc-sysctl-d\") pod \"tuned-hskxk\" (UID: \"ed08bc51-70e6-4658-adb3-ed36f5176552\") " pod="openshift-cluster-node-tuning-operator/tuned-hskxk"
Oct 30 21:41:46.951592 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.951049    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pd6bv\" (UniqueName: \"kubernetes.io/projected/ed08bc51-70e6-4658-adb3-ed36f5176552-kube-api-access-pd6bv\") pod \"tuned-hskxk\" (UID: \"ed08bc51-70e6-4658-adb3-ed36f5176552\") " pod="openshift-cluster-node-tuning-operator/tuned-hskxk"
Oct 30 21:41:46.951592 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.951072    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/538d5103-1c92-48f1-acc7-f90956848727-host-kubelet\") pod \"ovnkube-node-22ldv\" (UID: \"538d5103-1c92-48f1-acc7-f90956848727\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ldv"
Oct 30 21:41:46.952455 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.951095    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/538d5103-1c92-48f1-acc7-f90956848727-host-run-ovn-kubernetes\") pod \"ovnkube-node-22ldv\" (UID: \"538d5103-1c92-48f1-acc7-f90956848727\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ldv"
Oct 30 21:41:46.952455 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.951121    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1f3b6a23-d7ea-4ce4-b1e8-68133c4b4ebc-cni-binary-copy\") pod \"multus-additional-cni-plugins-b47v2\" (UID: \"1f3b6a23-d7ea-4ce4-b1e8-68133c4b4ebc\") " pod="openshift-multus/multus-additional-cni-plugins-b47v2"
Oct 30 21:41:46.952455 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.951126    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1f3b6a23-d7ea-4ce4-b1e8-68133c4b4ebc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-b47v2\" (UID: \"1f3b6a23-d7ea-4ce4-b1e8-68133c4b4ebc\") " pod="openshift-multus/multus-additional-cni-plugins-b47v2"
Oct 30 21:41:46.952455 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.951167    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0929b9f6-12ec-4f4d-8ba3-5cef9defba15-host-slash\") pod \"iptables-alerter-wbq4l\" (UID: \"0929b9f6-12ec-4f4d-8ba3-5cef9defba15\") " pod="openshift-network-operator/iptables-alerter-wbq4l"
Oct 30 21:41:46.952455 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.951182    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/538d5103-1c92-48f1-acc7-f90956848727-host-run-ovn-kubernetes\") pod \"ovnkube-node-22ldv\" (UID: \"538d5103-1c92-48f1-acc7-f90956848727\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ldv"
Oct 30 21:41:46.952455 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.951225    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4427f59d-8003-440f-9646-a46b176cc4a7-registration-dir\") pod \"aws-ebs-csi-driver-node-qk9dg\" (UID: \"4427f59d-8003-440f-9646-a46b176cc4a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qk9dg"
Oct 30 21:41:46.952455 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.951243    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ed08bc51-70e6-4658-adb3-ed36f5176552-sys\") pod \"tuned-hskxk\" (UID: \"ed08bc51-70e6-4658-adb3-ed36f5176552\") " pod="openshift-cluster-node-tuning-operator/tuned-hskxk"
Oct 30 21:41:46.952455 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.951258    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ed08bc51-70e6-4658-adb3-ed36f5176552-var-lib-kubelet\") pod \"tuned-hskxk\" (UID: \"ed08bc51-70e6-4658-adb3-ed36f5176552\") " pod="openshift-cluster-node-tuning-operator/tuned-hskxk"
Oct 30 21:41:46.952455 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.951272    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/238ff7d8-7da8-4cd8-9d45-deedf615211e-agent-certs\") pod \"konnectivity-agent-sztmx\" (UID: \"238ff7d8-7da8-4cd8-9d45-deedf615211e\") " pod="kube-system/konnectivity-agent-sztmx"
Oct 30 21:41:46.952455 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.951288    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-579v9\" (UniqueName: \"kubernetes.io/projected/3784e680-5efe-4ffb-ad71-956d05b57dce-kube-api-access-579v9\") pod \"network-check-target-979r9\" (UID: \"3784e680-5efe-4ffb-ad71-956d05b57dce\") " pod="openshift-network-diagnostics/network-check-target-979r9"
Oct 30 21:41:46.952455 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.951308    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1f3b6a23-d7ea-4ce4-b1e8-68133c4b4ebc-system-cni-dir\") pod \"multus-additional-cni-plugins-b47v2\" (UID: \"1f3b6a23-d7ea-4ce4-b1e8-68133c4b4ebc\") " pod="openshift-multus/multus-additional-cni-plugins-b47v2"
Oct 30 21:41:46.952455 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.951326    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/538d5103-1c92-48f1-acc7-f90956848727-ovnkube-script-lib\") pod \"ovnkube-node-22ldv\" (UID: \"538d5103-1c92-48f1-acc7-f90956848727\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ldv"
Oct 30 21:41:46.952455 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.951342    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a2e3b25e-b6dc-46ef-86e8-4e70f9dad200-multus-daemon-config\") pod \"multus-4nqkj\" (UID: \"a2e3b25e-b6dc-46ef-86e8-4e70f9dad200\") " pod="openshift-multus/multus-4nqkj"
Oct 30 21:41:46.952455 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.951345    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4427f59d-8003-440f-9646-a46b176cc4a7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qk9dg\" (UID: \"4427f59d-8003-440f-9646-a46b176cc4a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qk9dg"
Oct 30 21:41:46.952455 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.951349    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/538d5103-1c92-48f1-acc7-f90956848727-host-kubelet\") pod \"ovnkube-node-22ldv\" (UID: \"538d5103-1c92-48f1-acc7-f90956848727\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ldv"
Oct 30 21:41:46.952455 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.951370    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4427f59d-8003-440f-9646-a46b176cc4a7-etc-selinux\") pod \"aws-ebs-csi-driver-node-qk9dg\" (UID: \"4427f59d-8003-440f-9646-a46b176cc4a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qk9dg"
Oct 30 21:41:46.952455 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.951402    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a2e3b25e-b6dc-46ef-86e8-4e70f9dad200-etc-kubernetes\") pod \"multus-4nqkj\" (UID: \"a2e3b25e-b6dc-46ef-86e8-4e70f9dad200\") " pod="openshift-multus/multus-4nqkj"
Oct 30 21:41:46.953274 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.951416    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b227ad6a-9e59-415e-b7e6-f7c25e988abb-serviceca\") pod \"node-ca-x8gnj\" (UID: \"b227ad6a-9e59-415e-b7e6-f7c25e988abb\") " pod="openshift-image-registry/node-ca-x8gnj"
Oct 30 21:41:46.953274 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.951431    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fe41c70b-7e2c-432f-8389-0282d6fdb332-kubelet-config\") pod \"global-pull-secret-syncer-6wbw4\" (UID: \"fe41c70b-7e2c-432f-8389-0282d6fdb332\") " pod="kube-system/global-pull-secret-syncer-6wbw4"
Oct 30 21:41:46.953274 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.951445    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/538d5103-1c92-48f1-acc7-f90956848727-host-run-netns\") pod \"ovnkube-node-22ldv\" (UID: \"538d5103-1c92-48f1-acc7-f90956848727\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ldv"
Oct 30 21:41:46.953274 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.951460    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a2e3b25e-b6dc-46ef-86e8-4e70f9dad200-multus-cni-dir\") pod \"multus-4nqkj\" (UID: \"a2e3b25e-b6dc-46ef-86e8-4e70f9dad200\") " pod="openshift-multus/multus-4nqkj"
Oct 30 21:41:46.953274 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.951475    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a2e3b25e-b6dc-46ef-86e8-4e70f9dad200-cnibin\") pod \"multus-4nqkj\" (UID: \"a2e3b25e-b6dc-46ef-86e8-4e70f9dad200\") " pod="openshift-multus/multus-4nqkj"
Oct 30 21:41:46.953274 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.951490    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a2e3b25e-b6dc-46ef-86e8-4e70f9dad200-host-var-lib-cni-bin\") pod \"multus-4nqkj\" (UID: \"a2e3b25e-b6dc-46ef-86e8-4e70f9dad200\") " pod="openshift-multus/multus-4nqkj"
Oct 30 21:41:46.953274 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.951505    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gq9m\" (UniqueName: \"kubernetes.io/projected/1f3b6a23-d7ea-4ce4-b1e8-68133c4b4ebc-kube-api-access-7gq9m\") pod \"multus-additional-cni-plugins-b47v2\" (UID: \"1f3b6a23-d7ea-4ce4-b1e8-68133c4b4ebc\") " pod="openshift-multus/multus-additional-cni-plugins-b47v2"
Oct 30 21:41:46.953274 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.951521    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2205e8e6-f038-47fa-b393-0abe4d622e03-metrics-certs\") pod \"network-metrics-daemon-q6tjc\" (UID: \"2205e8e6-f038-47fa-b393-0abe4d622e03\") " pod="openshift-multus/network-metrics-daemon-q6tjc"
Oct 30 21:41:46.953274 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.951560    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swjcw\" (UniqueName: \"kubernetes.io/projected/0929b9f6-12ec-4f4d-8ba3-5cef9defba15-kube-api-access-swjcw\") pod \"iptables-alerter-wbq4l\" (UID: \"0929b9f6-12ec-4f4d-8ba3-5cef9defba15\") " pod="openshift-network-operator/iptables-alerter-wbq4l"
Oct 30 21:41:46.953274 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.951574    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/538d5103-1c92-48f1-acc7-f90956848727-ovnkube-config\") pod \"ovnkube-node-22ldv\" (UID: \"538d5103-1c92-48f1-acc7-f90956848727\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ldv"
Oct 30 21:41:46.953274 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.951580    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ed08bc51-70e6-4658-adb3-ed36f5176552-sys\") pod \"tuned-hskxk\" (UID: \"ed08bc51-70e6-4658-adb3-ed36f5176552\") " pod="openshift-cluster-node-tuning-operator/tuned-hskxk"
Oct 30 21:41:46.953274 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.951591    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a2e3b25e-b6dc-46ef-86e8-4e70f9dad200-host-var-lib-kubelet\") pod \"multus-4nqkj\" (UID: \"a2e3b25e-b6dc-46ef-86e8-4e70f9dad200\") " pod="openshift-multus/multus-4nqkj"
Oct 30 21:41:46.953274 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.951608    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ed08bc51-70e6-4658-adb3-ed36f5176552-etc-systemd\") pod \"tuned-hskxk\" (UID: \"ed08bc51-70e6-4658-adb3-ed36f5176552\") " pod="openshift-cluster-node-tuning-operator/tuned-hskxk"
Oct 30 21:41:46.953274 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.951626    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ed08bc51-70e6-4658-adb3-ed36f5176552-var-lib-kubelet\") pod \"tuned-hskxk\" (UID: \"ed08bc51-70e6-4658-adb3-ed36f5176552\") " pod="openshift-cluster-node-tuning-operator/tuned-hskxk"
Oct 30 21:41:46.953274 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.951634    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/71ddf455-cbf7-49a6-9773-30f7e4997a39-hosts-file\") pod \"node-resolver-2wsrr\" (UID: \"71ddf455-cbf7-49a6-9773-30f7e4997a39\") " pod="openshift-dns/node-resolver-2wsrr"
Oct 30 21:41:46.953274 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.951650    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/538d5103-1c92-48f1-acc7-f90956848727-ovn-node-metrics-cert\") pod \"ovnkube-node-22ldv\" (UID: \"538d5103-1c92-48f1-acc7-f90956848727\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ldv"
Oct 30 21:41:46.953274 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.951674    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a2e3b25e-b6dc-46ef-86e8-4e70f9dad200-host-run-multus-certs\") pod \"multus-4nqkj\" (UID: \"a2e3b25e-b6dc-46ef-86e8-4e70f9dad200\") " pod="openshift-multus/multus-4nqkj"
Oct 30 21:41:46.953914 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.951701    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4427f59d-8003-440f-9646-a46b176cc4a7-sys-fs\") pod \"aws-ebs-csi-driver-node-qk9dg\" (UID: \"4427f59d-8003-440f-9646-a46b176cc4a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qk9dg"
Oct 30 21:41:46.953914 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.951711    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1f3b6a23-d7ea-4ce4-b1e8-68133c4b4ebc-cni-binary-copy\") pod \"multus-additional-cni-plugins-b47v2\" (UID: \"1f3b6a23-d7ea-4ce4-b1e8-68133c4b4ebc\") " pod="openshift-multus/multus-additional-cni-plugins-b47v2"
Oct 30 21:41:46.953914 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.951716    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ed08bc51-70e6-4658-adb3-ed36f5176552-run\") pod \"tuned-hskxk\" (UID: \"ed08bc51-70e6-4658-adb3-ed36f5176552\") " pod="openshift-cluster-node-tuning-operator/tuned-hskxk"
Oct 30 21:41:46.953914 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.951746    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ed08bc51-70e6-4658-adb3-ed36f5176552-run\") pod \"tuned-hskxk\" (UID: \"ed08bc51-70e6-4658-adb3-ed36f5176552\") " pod="openshift-cluster-node-tuning-operator/tuned-hskxk"
Oct 30 21:41:46.953914 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.951764    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbwzr\" (UniqueName: \"kubernetes.io/projected/2205e8e6-f038-47fa-b393-0abe4d622e03-kube-api-access-rbwzr\") pod \"network-metrics-daemon-q6tjc\" (UID: \"2205e8e6-f038-47fa-b393-0abe4d622e03\") " pod="openshift-multus/network-metrics-daemon-q6tjc"
Oct 30 21:41:46.953914 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.951779    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4427f59d-8003-440f-9646-a46b176cc4a7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qk9dg\" (UID: \"4427f59d-8003-440f-9646-a46b176cc4a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qk9dg"
Oct 30 21:41:46.953914 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.951795    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/538d5103-1c92-48f1-acc7-f90956848727-var-lib-openvswitch\") pod \"ovnkube-node-22ldv\" (UID: \"538d5103-1c92-48f1-acc7-f90956848727\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ldv"
Oct 30 21:41:46.953914 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.951821    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/538d5103-1c92-48f1-acc7-f90956848727-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-22ldv\" (UID: \"538d5103-1c92-48f1-acc7-f90956848727\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ldv"
Oct 30 21:41:46.953914 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.951834    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4427f59d-8003-440f-9646-a46b176cc4a7-etc-selinux\") pod \"aws-ebs-csi-driver-node-qk9dg\" (UID: \"4427f59d-8003-440f-9646-a46b176cc4a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qk9dg"
Oct 30 21:41:46.953914 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.951847    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a2e3b25e-b6dc-46ef-86e8-4e70f9dad200-host-run-netns\") pod \"multus-4nqkj\" (UID: \"a2e3b25e-b6dc-46ef-86e8-4e70f9dad200\") " pod="openshift-multus/multus-4nqkj"
Oct 30 21:41:46.953914 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.951861    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a2e3b25e-b6dc-46ef-86e8-4e70f9dad200-etc-kubernetes\") pod \"multus-4nqkj\" (UID: \"a2e3b25e-b6dc-46ef-86e8-4e70f9dad200\") " pod="openshift-multus/multus-4nqkj"
Oct 30 21:41:46.953914 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.951871    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1f3b6a23-d7ea-4ce4-b1e8-68133c4b4ebc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-b47v2\" (UID: \"1f3b6a23-d7ea-4ce4-b1e8-68133c4b4ebc\") " pod="openshift-multus/multus-additional-cni-plugins-b47v2"
Oct 30 21:41:46.953914 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.951920    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71ddf455-cbf7-49a6-9773-30f7e4997a39-tmp-dir\") pod \"node-resolver-2wsrr\" (UID: \"71ddf455-cbf7-49a6-9773-30f7e4997a39\") " pod="openshift-dns/node-resolver-2wsrr"
Oct 30 21:41:46.953914 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.951934    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/538d5103-1c92-48f1-acc7-f90956848727-ovnkube-script-lib\") pod \"ovnkube-node-22ldv\" (UID: \"538d5103-1c92-48f1-acc7-f90956848727\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ldv"
Oct 30 21:41:46.953914 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.951950    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1f3b6a23-d7ea-4ce4-b1e8-68133c4b4ebc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-b47v2\" (UID: \"1f3b6a23-d7ea-4ce4-b1e8-68133c4b4ebc\") " pod="openshift-multus/multus-additional-cni-plugins-b47v2"
Oct 30 21:41:46.953914 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.950893    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ed08bc51-70e6-4658-adb3-ed36f5176552-etc-modprobe-d\") pod \"tuned-hskxk\" (UID: \"ed08bc51-70e6-4658-adb3-ed36f5176552\") " pod="openshift-cluster-node-tuning-operator/tuned-hskxk"
Oct 30 21:41:46.953914 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.951974    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/538d5103-1c92-48f1-acc7-f90956848727-run-systemd\") pod \"ovnkube-node-22ldv\" (UID: \"538d5103-1c92-48f1-acc7-f90956848727\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ldv"
Oct 30 21:41:46.954374 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.951975    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1f3b6a23-d7ea-4ce4-b1e8-68133c4b4ebc-system-cni-dir\") pod \"multus-additional-cni-plugins-b47v2\" (UID: \"1f3b6a23-d7ea-4ce4-b1e8-68133c4b4ebc\") " pod="openshift-multus/multus-additional-cni-plugins-b47v2"
Oct 30 21:41:46.954374 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.951765    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/238ff7d8-7da8-4cd8-9d45-deedf615211e-konnectivity-ca\") pod \"konnectivity-agent-sztmx\" (UID: \"238ff7d8-7da8-4cd8-9d45-deedf615211e\") " pod="kube-system/konnectivity-agent-sztmx"
Oct 30 21:41:46.954374 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.952020    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/538d5103-1c92-48f1-acc7-f90956848727-log-socket\") pod \"ovnkube-node-22ldv\" (UID: \"538d5103-1c92-48f1-acc7-f90956848727\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ldv"
Oct 30 21:41:46.954374 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.952045    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/538d5103-1c92-48f1-acc7-f90956848727-host-cni-netd\") pod \"ovnkube-node-22ldv\" (UID: \"538d5103-1c92-48f1-acc7-f90956848727\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ldv"
Oct 30 21:41:46.954374 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.952045    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fe41c70b-7e2c-432f-8389-0282d6fdb332-kubelet-config\") pod \"global-pull-secret-syncer-6wbw4\" (UID: \"fe41c70b-7e2c-432f-8389-0282d6fdb332\") " pod="kube-system/global-pull-secret-syncer-6wbw4"
Oct 30 21:41:46.954374 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.952081    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/538d5103-1c92-48f1-acc7-f90956848727-host-run-netns\") pod \"ovnkube-node-22ldv\" (UID: \"538d5103-1c92-48f1-acc7-f90956848727\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ldv"
Oct 30 21:41:46.954374 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.952110    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a2e3b25e-b6dc-46ef-86e8-4e70f9dad200-hostroot\") pod \"multus-4nqkj\" (UID: \"a2e3b25e-b6dc-46ef-86e8-4e70f9dad200\") " pod="openshift-multus/multus-4nqkj"
Oct 30 21:41:46.954374 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.952111    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/538d5103-1c92-48f1-acc7-f90956848727-env-overrides\") pod \"ovnkube-node-22ldv\" (UID: \"538d5103-1c92-48f1-acc7-f90956848727\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ldv"
Oct 30 21:41:46.954374 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.952135    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ed08bc51-70e6-4658-adb3-ed36f5176552-host\") pod \"tuned-hskxk\" (UID: \"ed08bc51-70e6-4658-adb3-ed36f5176552\") " pod="openshift-cluster-node-tuning-operator/tuned-hskxk"
Oct 30 21:41:46.954374 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.952157    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b227ad6a-9e59-415e-b7e6-f7c25e988abb-host\") pod \"node-ca-x8gnj\" (UID: \"b227ad6a-9e59-415e-b7e6-f7c25e988abb\") " pod="openshift-image-registry/node-ca-x8gnj"
Oct 30 21:41:46.954374 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.952169    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/538d5103-1c92-48f1-acc7-f90956848727-log-socket\") pod \"ovnkube-node-22ldv\" (UID: \"538d5103-1c92-48f1-acc7-f90956848727\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ldv"
Oct 30 21:41:46.954374 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.952181    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0929b9f6-12ec-4f4d-8ba3-5cef9defba15-iptables-alerter-script\") pod \"iptables-alerter-wbq4l\" (UID: \"0929b9f6-12ec-4f4d-8ba3-5cef9defba15\") " pod="openshift-network-operator/iptables-alerter-wbq4l"
Oct 30 21:41:46.954374 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.952208    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/538d5103-1c92-48f1-acc7-f90956848727-host-slash\") pod \"ovnkube-node-22ldv\" (UID: \"538d5103-1c92-48f1-acc7-f90956848727\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ldv"
Oct 30 21:41:46.954374 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.952228    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a2e3b25e-b6dc-46ef-86e8-4e70f9dad200-multus-cni-dir\") pod \"multus-4nqkj\" (UID: \"a2e3b25e-b6dc-46ef-86e8-4e70f9dad200\") " pod="openshift-multus/multus-4nqkj"
Oct 30 21:41:46.954374 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.952232    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8b7v\" (UniqueName: \"kubernetes.io/projected/a2e3b25e-b6dc-46ef-86e8-4e70f9dad200-kube-api-access-p8b7v\") pod \"multus-4nqkj\" (UID: \"a2e3b25e-b6dc-46ef-86e8-4e70f9dad200\") " pod="openshift-multus/multus-4nqkj"
Oct 30 21:41:46.954374 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.952263    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4427f59d-8003-440f-9646-a46b176cc4a7-device-dir\") pod \"aws-ebs-csi-driver-node-qk9dg\" (UID: \"4427f59d-8003-440f-9646-a46b176cc4a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qk9dg"
Oct 30 21:41:46.954374 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.952277    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b227ad6a-9e59-415e-b7e6-f7c25e988abb-serviceca\") pod \"node-ca-x8gnj\" (UID: \"b227ad6a-9e59-415e-b7e6-f7c25e988abb\") " pod="openshift-image-registry/node-ca-x8gnj"
Oct 30 21:41:46.954374 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.952287    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ed08bc51-70e6-4658-adb3-ed36f5176552-etc-tuned\") pod \"tuned-hskxk\" (UID: \"ed08bc51-70e6-4658-adb3-ed36f5176552\") " pod="openshift-cluster-node-tuning-operator/tuned-hskxk"
Oct 30 21:41:46.954974 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.952320    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fe41c70b-7e2c-432f-8389-0282d6fdb332-dbus\") pod \"global-pull-secret-syncer-6wbw4\" (UID: \"fe41c70b-7e2c-432f-8389-0282d6fdb332\") " pod="kube-system/global-pull-secret-syncer-6wbw4"
Oct 30 21:41:46.954974 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.952340    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a2e3b25e-b6dc-46ef-86e8-4e70f9dad200-host-var-lib-kubelet\") pod \"multus-4nqkj\" (UID: \"a2e3b25e-b6dc-46ef-86e8-4e70f9dad200\") " pod="openshift-multus/multus-4nqkj"
Oct 30 21:41:46.954974 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.952348    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fe41c70b-7e2c-432f-8389-0282d6fdb332-original-pull-secret\") pod \"global-pull-secret-syncer-6wbw4\" (UID: \"fe41c70b-7e2c-432f-8389-0282d6fdb332\") " pod="kube-system/global-pull-secret-syncer-6wbw4"
Oct 30 21:41:46.954974 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.952373    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/538d5103-1c92-48f1-acc7-f90956848727-etc-openvswitch\") pod \"ovnkube-node-22ldv\" (UID: \"538d5103-1c92-48f1-acc7-f90956848727\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ldv"
Oct 30 21:41:46.954974 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.952377    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/538d5103-1c92-48f1-acc7-f90956848727-host-cni-netd\") pod \"ovnkube-node-22ldv\" (UID: \"538d5103-1c92-48f1-acc7-f90956848727\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ldv"
Oct 30 21:41:46.954974 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.952394    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ed08bc51-70e6-4658-adb3-ed36f5176552-etc-systemd\") pod \"tuned-hskxk\" (UID: \"ed08bc51-70e6-4658-adb3-ed36f5176552\") " pod="openshift-cluster-node-tuning-operator/tuned-hskxk"
Oct 30 21:41:46.954974 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.952398    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/538d5103-1c92-48f1-acc7-f90956848727-node-log\") pod \"ovnkube-node-22ldv\" (UID: \"538d5103-1c92-48f1-acc7-f90956848727\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ldv"
Oct 30 21:41:46.954974 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.952430    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a2e3b25e-b6dc-46ef-86e8-4e70f9dad200-host-run-netns\") pod \"multus-4nqkj\" (UID: \"a2e3b25e-b6dc-46ef-86e8-4e70f9dad200\") " pod="openshift-multus/multus-4nqkj"
Oct 30 21:41:46.954974 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.952448    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/71ddf455-cbf7-49a6-9773-30f7e4997a39-hosts-file\") pod \"node-resolver-2wsrr\" (UID: \"71ddf455-cbf7-49a6-9773-30f7e4997a39\") " pod="openshift-dns/node-resolver-2wsrr"
Oct 30 21:41:46.954974 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.952451    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a2e3b25e-b6dc-46ef-86e8-4e70f9dad200-cnibin\") pod \"multus-4nqkj\" (UID: \"a2e3b25e-b6dc-46ef-86e8-4e70f9dad200\") " pod="openshift-multus/multus-4nqkj"
Oct 30 21:41:46.954974 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.952457    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1f3b6a23-d7ea-4ce4-b1e8-68133c4b4ebc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-b47v2\" (UID: \"1f3b6a23-d7ea-4ce4-b1e8-68133c4b4ebc\") " pod="openshift-multus/multus-additional-cni-plugins-b47v2"
Oct 30 21:41:46.954974 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.952466    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/538d5103-1c92-48f1-acc7-f90956848727-node-log\") pod \"ovnkube-node-22ldv\" (UID: \"538d5103-1c92-48f1-acc7-f90956848727\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ldv"
Oct 30 21:41:46.954974 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.952470    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1f3b6a23-d7ea-4ce4-b1e8-68133c4b4ebc-cnibin\") pod \"multus-additional-cni-plugins-b47v2\" (UID: \"1f3b6a23-d7ea-4ce4-b1e8-68133c4b4ebc\") " pod="openshift-multus/multus-additional-cni-plugins-b47v2"
Oct 30 21:41:46.954974 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:41:46.952495    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:41:46.954974 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.952497    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrpz5\" (UniqueName: \"kubernetes.io/projected/538d5103-1c92-48f1-acc7-f90956848727-kube-api-access-hrpz5\") pod \"ovnkube-node-22ldv\" (UID: \"538d5103-1c92-48f1-acc7-f90956848727\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ldv"
Oct 30 21:41:46.954974 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.952502    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a2e3b25e-b6dc-46ef-86e8-4e70f9dad200-host-run-multus-certs\") pod \"multus-4nqkj\" (UID: \"a2e3b25e-b6dc-46ef-86e8-4e70f9dad200\") " pod="openshift-multus/multus-4nqkj"
Oct 30 21:41:46.954974 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.952517    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/538d5103-1c92-48f1-acc7-f90956848727-etc-openvswitch\") pod \"ovnkube-node-22ldv\" (UID: \"538d5103-1c92-48f1-acc7-f90956848727\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ldv"
Oct 30 21:41:46.955539 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:41:46.952571    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2205e8e6-f038-47fa-b393-0abe4d622e03-metrics-certs podName:2205e8e6-f038-47fa-b393-0abe4d622e03 nodeName:}" failed. No retries permitted until 2026-10-30 21:41:47.452552122 +0000 UTC m=+2.262882149 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2205e8e6-f038-47fa-b393-0abe4d622e03-metrics-certs") pod "network-metrics-daemon-q6tjc" (UID: "2205e8e6-f038-47fa-b393-0abe4d622e03") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:41:46.955539 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.952589    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fe41c70b-7e2c-432f-8389-0282d6fdb332-dbus\") pod \"global-pull-secret-syncer-6wbw4\" (UID: \"fe41c70b-7e2c-432f-8389-0282d6fdb332\") " pod="kube-system/global-pull-secret-syncer-6wbw4"
Oct 30 21:41:46.955539 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.952610    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/538d5103-1c92-48f1-acc7-f90956848727-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-22ldv\" (UID: \"538d5103-1c92-48f1-acc7-f90956848727\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ldv"
Oct 30 21:41:46.955539 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:41:46.952663    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 21:41:46.955539 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:41:46.952708    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe41c70b-7e2c-432f-8389-0282d6fdb332-original-pull-secret podName:fe41c70b-7e2c-432f-8389-0282d6fdb332 nodeName:}" failed. No retries permitted until 2026-10-30 21:41:47.452691231 +0000 UTC m=+2.263021246 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fe41c70b-7e2c-432f-8389-0282d6fdb332-original-pull-secret") pod "global-pull-secret-syncer-6wbw4" (UID: "fe41c70b-7e2c-432f-8389-0282d6fdb332") : object "kube-system"/"original-pull-secret" not registered
Oct 30 21:41:46.955539 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.952742    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1f3b6a23-d7ea-4ce4-b1e8-68133c4b4ebc-cnibin\") pod \"multus-additional-cni-plugins-b47v2\" (UID: \"1f3b6a23-d7ea-4ce4-b1e8-68133c4b4ebc\") " pod="openshift-multus/multus-additional-cni-plugins-b47v2"
Oct 30 21:41:46.955539 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.952753    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71ddf455-cbf7-49a6-9773-30f7e4997a39-tmp-dir\") pod \"node-resolver-2wsrr\" (UID: \"71ddf455-cbf7-49a6-9773-30f7e4997a39\") " pod="openshift-dns/node-resolver-2wsrr"
Oct 30 21:41:46.955539 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.952792    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4427f59d-8003-440f-9646-a46b176cc4a7-device-dir\") pod \"aws-ebs-csi-driver-node-qk9dg\" (UID: \"4427f59d-8003-440f-9646-a46b176cc4a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qk9dg"
Oct 30 21:41:46.955539 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.952809    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/538d5103-1c92-48f1-acc7-f90956848727-run-systemd\") pod \"ovnkube-node-22ldv\" (UID: \"538d5103-1c92-48f1-acc7-f90956848727\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ldv"
Oct 30 21:41:46.955539 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.952837    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a2e3b25e-b6dc-46ef-86e8-4e70f9dad200-host-var-lib-cni-bin\") pod \"multus-4nqkj\" (UID: \"a2e3b25e-b6dc-46ef-86e8-4e70f9dad200\") " pod="openshift-multus/multus-4nqkj"
Oct 30 21:41:46.955539 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.952852    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a2e3b25e-b6dc-46ef-86e8-4e70f9dad200-system-cni-dir\") pod \"multus-4nqkj\" (UID: \"a2e3b25e-b6dc-46ef-86e8-4e70f9dad200\") " pod="openshift-multus/multus-4nqkj"
Oct 30 21:41:46.955539 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.952877    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b227ad6a-9e59-415e-b7e6-f7c25e988abb-host\") pod \"node-ca-x8gnj\" (UID: \"b227ad6a-9e59-415e-b7e6-f7c25e988abb\") " pod="openshift-image-registry/node-ca-x8gnj"
Oct 30 21:41:46.955539 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.952882    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ed08bc51-70e6-4658-adb3-ed36f5176552-lib-modules\") pod \"tuned-hskxk\" (UID: \"ed08bc51-70e6-4658-adb3-ed36f5176552\") " pod="openshift-cluster-node-tuning-operator/tuned-hskxk"
Oct 30 21:41:46.955539 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.952909    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hkfh\" (UniqueName: \"kubernetes.io/projected/71ddf455-cbf7-49a6-9773-30f7e4997a39-kube-api-access-7hkfh\") pod \"node-resolver-2wsrr\" (UID: \"71ddf455-cbf7-49a6-9773-30f7e4997a39\") " pod="openshift-dns/node-resolver-2wsrr"
Oct 30 21:41:46.955539 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.952916    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a2e3b25e-b6dc-46ef-86e8-4e70f9dad200-hostroot\") pod \"multus-4nqkj\" (UID: \"a2e3b25e-b6dc-46ef-86e8-4e70f9dad200\") " pod="openshift-multus/multus-4nqkj"
Oct 30 21:41:46.955539 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.953022    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ed08bc51-70e6-4658-adb3-ed36f5176552-lib-modules\") pod \"tuned-hskxk\" (UID: \"ed08bc51-70e6-4658-adb3-ed36f5176552\") " pod="openshift-cluster-node-tuning-operator/tuned-hskxk"
Oct 30 21:41:46.955539 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.953070    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a2e3b25e-b6dc-46ef-86e8-4e70f9dad200-system-cni-dir\") pod \"multus-4nqkj\" (UID: \"a2e3b25e-b6dc-46ef-86e8-4e70f9dad200\") " pod="openshift-multus/multus-4nqkj"
Oct 30 21:41:46.956000 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.953102    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1f3b6a23-d7ea-4ce4-b1e8-68133c4b4ebc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-b47v2\" (UID: \"1f3b6a23-d7ea-4ce4-b1e8-68133c4b4ebc\") " pod="openshift-multus/multus-additional-cni-plugins-b47v2"
Oct 30 21:41:46.956000 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.953114    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/538d5103-1c92-48f1-acc7-f90956848727-host-slash\") pod \"ovnkube-node-22ldv\" (UID: \"538d5103-1c92-48f1-acc7-f90956848727\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ldv"
Oct 30 21:41:46.956000 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.953102    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/538d5103-1c92-48f1-acc7-f90956848727-ovnkube-config\") pod \"ovnkube-node-22ldv\" (UID: \"538d5103-1c92-48f1-acc7-f90956848727\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ldv"
Oct 30 21:41:46.956000 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.953165    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4427f59d-8003-440f-9646-a46b176cc4a7-sys-fs\") pod \"aws-ebs-csi-driver-node-qk9dg\" (UID: \"4427f59d-8003-440f-9646-a46b176cc4a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qk9dg"
Oct 30 21:41:46.956000 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.953171    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ed08bc51-70e6-4658-adb3-ed36f5176552-host\") pod \"tuned-hskxk\" (UID: \"ed08bc51-70e6-4658-adb3-ed36f5176552\") " pod="openshift-cluster-node-tuning-operator/tuned-hskxk"
Oct 30 21:41:46.956000 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.953171    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/538d5103-1c92-48f1-acc7-f90956848727-var-lib-openvswitch\") pod \"ovnkube-node-22ldv\" (UID: \"538d5103-1c92-48f1-acc7-f90956848727\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ldv"
Oct 30 21:41:46.956000 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.953649    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0929b9f6-12ec-4f4d-8ba3-5cef9defba15-iptables-alerter-script\") pod \"iptables-alerter-wbq4l\" (UID: \"0929b9f6-12ec-4f4d-8ba3-5cef9defba15\") " pod="openshift-network-operator/iptables-alerter-wbq4l"
Oct 30 21:41:46.956000 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.954967    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ed08bc51-70e6-4658-adb3-ed36f5176552-tmp\") pod \"tuned-hskxk\" (UID: \"ed08bc51-70e6-4658-adb3-ed36f5176552\") " pod="openshift-cluster-node-tuning-operator/tuned-hskxk"
Oct 30 21:41:46.956000 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.955033    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ed08bc51-70e6-4658-adb3-ed36f5176552-etc-tuned\") pod \"tuned-hskxk\" (UID: \"ed08bc51-70e6-4658-adb3-ed36f5176552\") " pod="openshift-cluster-node-tuning-operator/tuned-hskxk"
Oct 30 21:41:46.956000 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.955163    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/538d5103-1c92-48f1-acc7-f90956848727-ovn-node-metrics-cert\") pod \"ovnkube-node-22ldv\" (UID: \"538d5103-1c92-48f1-acc7-f90956848727\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ldv"
Oct 30 21:41:46.956000 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:46.955187    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/238ff7d8-7da8-4cd8-9d45-deedf615211e-agent-certs\") pod \"konnectivity-agent-sztmx\" (UID: \"238ff7d8-7da8-4cd8-9d45-deedf615211e\") " pod="kube-system/konnectivity-agent-sztmx"
Oct 30 21:41:47.005056 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:41:47.005018    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 21:41:47.005056 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:41:47.005047    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 21:41:47.005056 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:41:47.005058    2580 projected.go:196] Error preparing data for projected volume kube-api-access-579v9 for pod openshift-network-diagnostics/network-check-target-979r9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:41:47.005330 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:41:47.005111    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3784e680-5efe-4ffb-ad71-956d05b57dce-kube-api-access-579v9 podName:3784e680-5efe-4ffb-ad71-956d05b57dce nodeName:}" failed. No retries permitted until 2026-10-30 21:41:47.505097759 +0000 UTC m=+2.315427773 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-579v9" (UniqueName: "kubernetes.io/projected/3784e680-5efe-4ffb-ad71-956d05b57dce-kube-api-access-579v9") pod "network-check-target-979r9" (UID: "3784e680-5efe-4ffb-ad71-956d05b57dce") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:41:47.006253 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:47.006228    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nc76p\" (UniqueName: \"kubernetes.io/projected/4427f59d-8003-440f-9646-a46b176cc4a7-kube-api-access-nc76p\") pod \"aws-ebs-csi-driver-node-qk9dg\" (UID: \"4427f59d-8003-440f-9646-a46b176cc4a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qk9dg"
Oct 30 21:41:47.007072 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:47.007045    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pd6bv\" (UniqueName: \"kubernetes.io/projected/ed08bc51-70e6-4658-adb3-ed36f5176552-kube-api-access-pd6bv\") pod \"tuned-hskxk\" (UID: \"ed08bc51-70e6-4658-adb3-ed36f5176552\") " pod="openshift-cluster-node-tuning-operator/tuned-hskxk"
Oct 30 21:41:47.008062 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:47.008041    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpbtv\" (UniqueName: \"kubernetes.io/projected/b227ad6a-9e59-415e-b7e6-f7c25e988abb-kube-api-access-qpbtv\") pod \"node-ca-x8gnj\" (UID: \"b227ad6a-9e59-415e-b7e6-f7c25e988abb\") " pod="openshift-image-registry/node-ca-x8gnj"
Oct 30 21:41:47.008137 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:47.008069    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbwzr\" (UniqueName: \"kubernetes.io/projected/2205e8e6-f038-47fa-b393-0abe4d622e03-kube-api-access-rbwzr\") pod \"network-metrics-daemon-q6tjc\" (UID: \"2205e8e6-f038-47fa-b393-0abe4d622e03\") " pod="openshift-multus/network-metrics-daemon-q6tjc"
Oct 30 21:41:47.009556 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:47.009518    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swjcw\" (UniqueName: \"kubernetes.io/projected/0929b9f6-12ec-4f4d-8ba3-5cef9defba15-kube-api-access-swjcw\") pod \"iptables-alerter-wbq4l\" (UID: \"0929b9f6-12ec-4f4d-8ba3-5cef9defba15\") " pod="openshift-network-operator/iptables-alerter-wbq4l"
Oct 30 21:41:47.011448 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:47.011430    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hkfh\" (UniqueName: \"kubernetes.io/projected/71ddf455-cbf7-49a6-9773-30f7e4997a39-kube-api-access-7hkfh\") pod \"node-resolver-2wsrr\" (UID: \"71ddf455-cbf7-49a6-9773-30f7e4997a39\") " pod="openshift-dns/node-resolver-2wsrr"
Oct 30 21:41:47.012870 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:47.012822    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrpz5\" (UniqueName: \"kubernetes.io/projected/538d5103-1c92-48f1-acc7-f90956848727-kube-api-access-hrpz5\") pod \"ovnkube-node-22ldv\" (UID: \"538d5103-1c92-48f1-acc7-f90956848727\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ldv"
Oct 30 21:41:47.012979 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:47.012963    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gq9m\" (UniqueName: \"kubernetes.io/projected/1f3b6a23-d7ea-4ce4-b1e8-68133c4b4ebc-kube-api-access-7gq9m\") pod \"multus-additional-cni-plugins-b47v2\" (UID: \"1f3b6a23-d7ea-4ce4-b1e8-68133c4b4ebc\") " pod="openshift-multus/multus-additional-cni-plugins-b47v2"
Oct 30 21:41:47.013950 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:47.013936    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8b7v\" (UniqueName: \"kubernetes.io/projected/a2e3b25e-b6dc-46ef-86e8-4e70f9dad200-kube-api-access-p8b7v\") pod \"multus-4nqkj\" (UID: \"a2e3b25e-b6dc-46ef-86e8-4e70f9dad200\") " pod="openshift-multus/multus-4nqkj"
Oct 30 21:41:47.097614 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:47.097586    2580 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:41:47.111746 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:47.111726    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wbq4l"
Oct 30 21:41:47.151812 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:47.151785    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-x8gnj"
Oct 30 21:41:47.157841 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:47.156652    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-22ldv"
Oct 30 21:41:47.161681 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:47.161661    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4nqkj"
Oct 30 21:41:47.185257 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:47.185239    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-b47v2"
Oct 30 21:41:47.192748 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:47.192731    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qk9dg"
Oct 30 21:41:47.206363 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:47.206347    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-hskxk"
Oct 30 21:41:47.211519 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:47.211505    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-2wsrr"
Oct 30 21:41:47.215995 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:47.215979    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-sztmx"
Oct 30 21:41:47.246594 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:47.246558    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0b65115e7be9fc3f67a9c81d1477e73.slice/crio-23d53277e0647a0a05ba276cdb89af8108237e92f2dae2b39a5e3f45d1452f89 WatchSource:0}: Error finding container 23d53277e0647a0a05ba276cdb89af8108237e92f2dae2b39a5e3f45d1452f89: Status 404 returned error can't find the container with id 23d53277e0647a0a05ba276cdb89af8108237e92f2dae2b39a5e3f45d1452f89
Oct 30 21:41:47.247193 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:47.247178    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod238ff7d8_7da8_4cd8_9d45_deedf615211e.slice/crio-4276dc97fd6bbdde5d02f3244ef251d2b85ea8637cc7e47e2d6a5d871405e8cc WatchSource:0}: Error finding container 4276dc97fd6bbdde5d02f3244ef251d2b85ea8637cc7e47e2d6a5d871405e8cc: Status 404 returned error can't find the container with id 4276dc97fd6bbdde5d02f3244ef251d2b85ea8637cc7e47e2d6a5d871405e8cc
Oct 30 21:41:47.248776 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:47.248756    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71ddf455_cbf7_49a6_9773_30f7e4997a39.slice/crio-a8dee9b06f8070907d581c085c189097c986b3347dddd64dfe1637cd79b4352c WatchSource:0}: Error finding container a8dee9b06f8070907d581c085c189097c986b3347dddd64dfe1637cd79b4352c: Status 404 returned error can't find the container with id a8dee9b06f8070907d581c085c189097c986b3347dddd64dfe1637cd79b4352c
Oct 30 21:41:47.249152 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:47.249131    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf62eb3c9c04acf0f549e44a10220f5a.slice/crio-76ef14d66ab9af8383c8ea83abcbfb7325220d8b7e326ef09c6a16fab7db51b2 WatchSource:0}: Error finding container 76ef14d66ab9af8383c8ea83abcbfb7325220d8b7e326ef09c6a16fab7db51b2: Status 404 returned error can't find the container with id 76ef14d66ab9af8383c8ea83abcbfb7325220d8b7e326ef09c6a16fab7db51b2
Oct 30 21:41:47.251621 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:47.251594    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 21:41:47.335591 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:47.335516    2580 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:41:47.456411 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:47.456381    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fe41c70b-7e2c-432f-8389-0282d6fdb332-original-pull-secret\") pod \"global-pull-secret-syncer-6wbw4\" (UID: \"fe41c70b-7e2c-432f-8389-0282d6fdb332\") " pod="kube-system/global-pull-secret-syncer-6wbw4"
Oct 30 21:41:47.456564 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:47.456440    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2205e8e6-f038-47fa-b393-0abe4d622e03-metrics-certs\") pod \"network-metrics-daemon-q6tjc\" (UID: \"2205e8e6-f038-47fa-b393-0abe4d622e03\") " pod="openshift-multus/network-metrics-daemon-q6tjc"
Oct 30 21:41:47.456564 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:41:47.456518    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:41:47.456642 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:41:47.456568    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2205e8e6-f038-47fa-b393-0abe4d622e03-metrics-certs podName:2205e8e6-f038-47fa-b393-0abe4d622e03 nodeName:}" failed. No retries permitted until 2026-10-30 21:41:48.456556708 +0000 UTC m=+3.266886721 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2205e8e6-f038-47fa-b393-0abe4d622e03-metrics-certs") pod "network-metrics-daemon-q6tjc" (UID: "2205e8e6-f038-47fa-b393-0abe4d622e03") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:41:47.456642 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:41:47.456518    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 21:41:47.456642 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:41:47.456602    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe41c70b-7e2c-432f-8389-0282d6fdb332-original-pull-secret podName:fe41c70b-7e2c-432f-8389-0282d6fdb332 nodeName:}" failed. No retries permitted until 2026-10-30 21:41:48.456593861 +0000 UTC m=+3.266923864 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fe41c70b-7e2c-432f-8389-0282d6fdb332-original-pull-secret") pod "global-pull-secret-syncer-6wbw4" (UID: "fe41c70b-7e2c-432f-8389-0282d6fdb332") : object "kube-system"/"original-pull-secret" not registered
Oct 30 21:41:47.556736 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:47.556708    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-579v9\" (UniqueName: \"kubernetes.io/projected/3784e680-5efe-4ffb-ad71-956d05b57dce-kube-api-access-579v9\") pod \"network-check-target-979r9\" (UID: \"3784e680-5efe-4ffb-ad71-956d05b57dce\") " pod="openshift-network-diagnostics/network-check-target-979r9"
Oct 30 21:41:47.557225 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:41:47.556846    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 21:41:47.557225 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:41:47.556859    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 21:41:47.557225 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:41:47.556868    2580 projected.go:196] Error preparing data for projected volume kube-api-access-579v9 for pod openshift-network-diagnostics/network-check-target-979r9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:41:47.557225 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:41:47.556907    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3784e680-5efe-4ffb-ad71-956d05b57dce-kube-api-access-579v9 podName:3784e680-5efe-4ffb-ad71-956d05b57dce nodeName:}" failed. No retries permitted until 2026-10-30 21:41:48.556896063 +0000 UTC m=+3.367226063 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-579v9" (UniqueName: "kubernetes.io/projected/3784e680-5efe-4ffb-ad71-956d05b57dce-kube-api-access-579v9") pod "network-check-target-979r9" (UID: "3784e680-5efe-4ffb-ad71-956d05b57dce") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:41:47.614289 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:47.614206    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-2wsrr" event={"ID":"71ddf455-cbf7-49a6-9773-30f7e4997a39","Type":"ContainerStarted","Data":"a8dee9b06f8070907d581c085c189097c986b3347dddd64dfe1637cd79b4352c"}
Oct 30 21:41:47.615191 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:47.615174    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-sztmx" event={"ID":"238ff7d8-7da8-4cd8-9d45-deedf615211e","Type":"ContainerStarted","Data":"4276dc97fd6bbdde5d02f3244ef251d2b85ea8637cc7e47e2d6a5d871405e8cc"}
Oct 30 21:41:47.616099 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:47.616081    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-60.us-west-2.compute.internal" event={"ID":"c0b65115e7be9fc3f67a9c81d1477e73","Type":"ContainerStarted","Data":"23d53277e0647a0a05ba276cdb89af8108237e92f2dae2b39a5e3f45d1452f89"}
Oct 30 21:41:47.617029 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:47.617002    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-60.us-west-2.compute.internal" event={"ID":"cf62eb3c9c04acf0f549e44a10220f5a","Type":"ContainerStarted","Data":"76ef14d66ab9af8383c8ea83abcbfb7325220d8b7e326ef09c6a16fab7db51b2"}
Oct 30 21:41:47.924846 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:47.924747    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 21:36:46 +0000 UTC" deadline="2028-04-29 16:04:08.859635937 +0000 UTC"
Oct 30 21:41:47.924846 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:47.924789    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13122h22m20.934850616s"
Oct 30 21:41:47.998753 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:47.998725    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0929b9f6_12ec_4f4d_8ba3_5cef9defba15.slice/crio-e94508390d125c0485323172c4f0b87688846973bfa23f1a4a6baa825ea13544 WatchSource:0}: Error finding container e94508390d125c0485323172c4f0b87688846973bfa23f1a4a6baa825ea13544: Status 404 returned error can't find the container with id e94508390d125c0485323172c4f0b87688846973bfa23f1a4a6baa825ea13544
Oct 30 21:41:48.033075 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:48.033045    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f3b6a23_d7ea_4ce4_b1e8_68133c4b4ebc.slice/crio-cd13c3e7b0eb8ad54aed3de1a1bb8c6b1edc99e29a2368dcc6d659c394591cfc WatchSource:0}: Error finding container cd13c3e7b0eb8ad54aed3de1a1bb8c6b1edc99e29a2368dcc6d659c394591cfc: Status 404 returned error can't find the container with id cd13c3e7b0eb8ad54aed3de1a1bb8c6b1edc99e29a2368dcc6d659c394591cfc
Oct 30 21:41:48.061432 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:48.061408    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod538d5103_1c92_48f1_acc7_f90956848727.slice/crio-be70abae34436026075497a968e2b20eea71e3e611aa0c26f209a31ef280dd9a WatchSource:0}: Error finding container be70abae34436026075497a968e2b20eea71e3e611aa0c26f209a31ef280dd9a: Status 404 returned error can't find the container with id be70abae34436026075497a968e2b20eea71e3e611aa0c26f209a31ef280dd9a
Oct 30 21:41:48.153724 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:48.153696    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2e3b25e_b6dc_46ef_86e8_4e70f9dad200.slice/crio-3d5727e52b59c14557e36a0b0d4361bfd0b88a8203162fddca3943e12183fcbc WatchSource:0}: Error finding container 3d5727e52b59c14557e36a0b0d4361bfd0b88a8203162fddca3943e12183fcbc: Status 404 returned error can't find the container with id 3d5727e52b59c14557e36a0b0d4361bfd0b88a8203162fddca3943e12183fcbc
Oct 30 21:41:48.173223 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:41:48.173202    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb227ad6a_9e59_415e_b7e6_f7c25e988abb.slice/crio-a0b977d201619f90b14be079450e973daf8fbde1644152a642ffd5b9ad30c498 WatchSource:0}: Error finding container a0b977d201619f90b14be079450e973daf8fbde1644152a642ffd5b9ad30c498: Status 404 returned error can't find the container with id a0b977d201619f90b14be079450e973daf8fbde1644152a642ffd5b9ad30c498
Oct 30 21:41:48.463366 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:48.463253    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2205e8e6-f038-47fa-b393-0abe4d622e03-metrics-certs\") pod \"network-metrics-daemon-q6tjc\" (UID: \"2205e8e6-f038-47fa-b393-0abe4d622e03\") " pod="openshift-multus/network-metrics-daemon-q6tjc"
Oct 30 21:41:48.463366 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:48.463326    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fe41c70b-7e2c-432f-8389-0282d6fdb332-original-pull-secret\") pod \"global-pull-secret-syncer-6wbw4\" (UID: \"fe41c70b-7e2c-432f-8389-0282d6fdb332\") " pod="kube-system/global-pull-secret-syncer-6wbw4"
Oct 30 21:41:48.463631 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:41:48.463446    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 21:41:48.463631 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:41:48.463494    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe41c70b-7e2c-432f-8389-0282d6fdb332-original-pull-secret podName:fe41c70b-7e2c-432f-8389-0282d6fdb332 nodeName:}" failed. No retries permitted until 2026-10-30 21:41:50.463481302 +0000 UTC m=+5.273811303 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fe41c70b-7e2c-432f-8389-0282d6fdb332-original-pull-secret") pod "global-pull-secret-syncer-6wbw4" (UID: "fe41c70b-7e2c-432f-8389-0282d6fdb332") : object "kube-system"/"original-pull-secret" not registered
Oct 30 21:41:48.463631 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:41:48.463551    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:41:48.463631 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:41:48.463580    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2205e8e6-f038-47fa-b393-0abe4d622e03-metrics-certs podName:2205e8e6-f038-47fa-b393-0abe4d622e03 nodeName:}" failed. No retries permitted until 2026-10-30 21:41:50.463569412 +0000 UTC m=+5.273899428 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2205e8e6-f038-47fa-b393-0abe4d622e03-metrics-certs") pod "network-metrics-daemon-q6tjc" (UID: "2205e8e6-f038-47fa-b393-0abe4d622e03") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:41:48.563749 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:48.563706    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-579v9\" (UniqueName: \"kubernetes.io/projected/3784e680-5efe-4ffb-ad71-956d05b57dce-kube-api-access-579v9\") pod \"network-check-target-979r9\" (UID: \"3784e680-5efe-4ffb-ad71-956d05b57dce\") " pod="openshift-network-diagnostics/network-check-target-979r9"
Oct 30 21:41:48.564296 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:41:48.563897    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 21:41:48.564296 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:41:48.563917    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 21:41:48.564296 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:41:48.563930    2580 projected.go:196] Error preparing data for projected volume kube-api-access-579v9 for pod openshift-network-diagnostics/network-check-target-979r9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:41:48.564296 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:41:48.563995    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3784e680-5efe-4ffb-ad71-956d05b57dce-kube-api-access-579v9 podName:3784e680-5efe-4ffb-ad71-956d05b57dce nodeName:}" failed. No retries permitted until 2026-10-30 21:41:50.563969273 +0000 UTC m=+5.374299281 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-579v9" (UniqueName: "kubernetes.io/projected/3784e680-5efe-4ffb-ad71-956d05b57dce-kube-api-access-579v9") pod "network-check-target-979r9" (UID: "3784e680-5efe-4ffb-ad71-956d05b57dce") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:41:48.611095 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:48.611062    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q6tjc"
Oct 30 21:41:48.611268 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:41:48.611189    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q6tjc" podUID="2205e8e6-f038-47fa-b393-0abe4d622e03"
Oct 30 21:41:48.611339 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:48.611289    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6wbw4"
Oct 30 21:41:48.611391 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:41:48.611367    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6wbw4" podUID="fe41c70b-7e2c-432f-8389-0282d6fdb332"
Oct 30 21:41:48.611441 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:48.611424    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-979r9"
Oct 30 21:41:48.611491 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:41:48.611468    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-979r9" podUID="3784e680-5efe-4ffb-ad71-956d05b57dce"
Oct 30 21:41:48.621734 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:48.621702    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-x8gnj" event={"ID":"b227ad6a-9e59-415e-b7e6-f7c25e988abb","Type":"ContainerStarted","Data":"a0b977d201619f90b14be079450e973daf8fbde1644152a642ffd5b9ad30c498"}
Oct 30 21:41:48.625153 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:48.624668    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-hskxk" event={"ID":"ed08bc51-70e6-4658-adb3-ed36f5176552","Type":"ContainerStarted","Data":"5d1a26f23fc7c9a064d213e02a05c2017bb01d15a3887ed2d76a2e54ca92c5e3"}
Oct 30 21:41:48.628440 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:48.627056    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qk9dg" event={"ID":"4427f59d-8003-440f-9646-a46b176cc4a7","Type":"ContainerStarted","Data":"e369a0f022d80f14a70bcfff6d7599bae3b42ee10574fb6f745e5f9bcc7fa2ad"}
Oct 30 21:41:48.637871 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:48.637839    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-22ldv" event={"ID":"538d5103-1c92-48f1-acc7-f90956848727","Type":"ContainerStarted","Data":"be70abae34436026075497a968e2b20eea71e3e611aa0c26f209a31ef280dd9a"}
Oct 30 21:41:48.639881 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:48.639861    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wbq4l" event={"ID":"0929b9f6-12ec-4f4d-8ba3-5cef9defba15","Type":"ContainerStarted","Data":"e94508390d125c0485323172c4f0b87688846973bfa23f1a4a6baa825ea13544"}
Oct 30 21:41:48.642286 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:48.642263    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4nqkj" event={"ID":"a2e3b25e-b6dc-46ef-86e8-4e70f9dad200","Type":"ContainerStarted","Data":"3d5727e52b59c14557e36a0b0d4361bfd0b88a8203162fddca3943e12183fcbc"}
Oct 30 21:41:48.643958 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:48.643935    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b47v2" event={"ID":"1f3b6a23-d7ea-4ce4-b1e8-68133c4b4ebc","Type":"ContainerStarted","Data":"cd13c3e7b0eb8ad54aed3de1a1bb8c6b1edc99e29a2368dcc6d659c394591cfc"}
Oct 30 21:41:48.925822 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:48.925678    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 21:36:46 +0000 UTC" deadline="2028-07-28 23:48:41.119396886 +0000 UTC"
Oct 30 21:41:48.925822 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:48.925770    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15290h6m52.193631385s"
Oct 30 21:41:50.482474 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:50.482437    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2205e8e6-f038-47fa-b393-0abe4d622e03-metrics-certs\") pod \"network-metrics-daemon-q6tjc\" (UID: \"2205e8e6-f038-47fa-b393-0abe4d622e03\") " pod="openshift-multus/network-metrics-daemon-q6tjc"
Oct 30 21:41:50.483042 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:50.482509    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fe41c70b-7e2c-432f-8389-0282d6fdb332-original-pull-secret\") pod \"global-pull-secret-syncer-6wbw4\" (UID: \"fe41c70b-7e2c-432f-8389-0282d6fdb332\") " pod="kube-system/global-pull-secret-syncer-6wbw4"
Oct 30 21:41:50.483042 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:41:50.482667    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 21:41:50.483042 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:41:50.482727    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe41c70b-7e2c-432f-8389-0282d6fdb332-original-pull-secret podName:fe41c70b-7e2c-432f-8389-0282d6fdb332 nodeName:}" failed. No retries permitted until 2026-10-30 21:41:54.482708828 +0000 UTC m=+9.293038841 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fe41c70b-7e2c-432f-8389-0282d6fdb332-original-pull-secret") pod "global-pull-secret-syncer-6wbw4" (UID: "fe41c70b-7e2c-432f-8389-0282d6fdb332") : object "kube-system"/"original-pull-secret" not registered
Oct 30 21:41:50.483213 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:41:50.483100    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:41:50.483213 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:41:50.483148    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2205e8e6-f038-47fa-b393-0abe4d622e03-metrics-certs podName:2205e8e6-f038-47fa-b393-0abe4d622e03 nodeName:}" failed. No retries permitted until 2026-10-30 21:41:54.483131456 +0000 UTC m=+9.293461473 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2205e8e6-f038-47fa-b393-0abe4d622e03-metrics-certs") pod "network-metrics-daemon-q6tjc" (UID: "2205e8e6-f038-47fa-b393-0abe4d622e03") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:41:50.583815 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:50.583756    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-579v9\" (UniqueName: \"kubernetes.io/projected/3784e680-5efe-4ffb-ad71-956d05b57dce-kube-api-access-579v9\") pod \"network-check-target-979r9\" (UID: \"3784e680-5efe-4ffb-ad71-956d05b57dce\") " pod="openshift-network-diagnostics/network-check-target-979r9"
Oct 30 21:41:50.583924 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:41:50.583887    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 21:41:50.583924 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:41:50.583900    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 21:41:50.583924 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:41:50.583908    2580 projected.go:196] Error preparing data for projected volume kube-api-access-579v9 for pod openshift-network-diagnostics/network-check-target-979r9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:41:50.584077 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:41:50.583951    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3784e680-5efe-4ffb-ad71-956d05b57dce-kube-api-access-579v9 podName:3784e680-5efe-4ffb-ad71-956d05b57dce nodeName:}" failed. No retries permitted until 2026-10-30 21:41:54.583939928 +0000 UTC m=+9.394269941 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-579v9" (UniqueName: "kubernetes.io/projected/3784e680-5efe-4ffb-ad71-956d05b57dce-kube-api-access-579v9") pod "network-check-target-979r9" (UID: "3784e680-5efe-4ffb-ad71-956d05b57dce") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:41:50.611230 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:50.611177    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6wbw4"
Oct 30 21:41:50.611384 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:41:50.611302    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6wbw4" podUID="fe41c70b-7e2c-432f-8389-0282d6fdb332"
Oct 30 21:41:50.611384 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:50.611374    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-979r9"
Oct 30 21:41:50.611489 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:41:50.611438    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-979r9" podUID="3784e680-5efe-4ffb-ad71-956d05b57dce"
Oct 30 21:41:50.611557 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:50.611493    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q6tjc"
Oct 30 21:41:50.611644 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:41:50.611593    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q6tjc" podUID="2205e8e6-f038-47fa-b393-0abe4d622e03"
Oct 30 21:41:51.660400 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:51.660169    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-60.us-west-2.compute.internal" event={"ID":"c0b65115e7be9fc3f67a9c81d1477e73","Type":"ContainerStarted","Data":"d1d30500380aa168c8228952650114a5604f5ca3e2c4d115b5dc145ee58e92e6"}
Oct 30 21:41:51.662795 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:51.662755    2580 generic.go:356] "Generic (PLEG): container finished" podID="cf62eb3c9c04acf0f549e44a10220f5a" containerID="25f6c7e10a3841baf6120fa239857b451f78ee39a794830fab2f601a27aad210" exitCode=0
Oct 30 21:41:51.662908 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:51.662806    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-60.us-west-2.compute.internal" event={"ID":"cf62eb3c9c04acf0f549e44a10220f5a","Type":"ContainerDied","Data":"25f6c7e10a3841baf6120fa239857b451f78ee39a794830fab2f601a27aad210"}
Oct 30 21:41:51.885886 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:51.885828    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-60.us-west-2.compute.internal" podStartSLOduration=5.885807576 podStartE2EDuration="5.885807576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:41:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:51.771514744 +0000 UTC m=+6.581844767" watchObservedRunningTime="2026-10-30 21:41:51.885807576 +0000 UTC m=+6.696137599"
Oct 30 21:41:52.610730 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:52.610652    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6wbw4"
Oct 30 21:41:52.610894 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:41:52.610790    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6wbw4" podUID="fe41c70b-7e2c-432f-8389-0282d6fdb332"
Oct 30 21:41:52.611199 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:52.611181    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-979r9"
Oct 30 21:41:52.611307 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:41:52.611278    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-979r9" podUID="3784e680-5efe-4ffb-ad71-956d05b57dce"
Oct 30 21:41:52.611365 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:52.611339    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q6tjc"
Oct 30 21:41:52.611455 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:41:52.611421    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q6tjc" podUID="2205e8e6-f038-47fa-b393-0abe4d622e03"
Oct 30 21:41:52.669242 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:52.669206    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-60.us-west-2.compute.internal" event={"ID":"cf62eb3c9c04acf0f549e44a10220f5a","Type":"ContainerStarted","Data":"5c04ec9c260142a34a460c97120a73f230417cb57d760dab3a98d0b1da29c2d2"}
Oct 30 21:41:52.673206 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:52.673141    2580 generic.go:356] "Generic (PLEG): container finished" podID="1f3b6a23-d7ea-4ce4-b1e8-68133c4b4ebc" containerID="9c3b30ec66ad103cdf19bbf59344440ab5f9f4677c7d202a77f05fb3a1191a22" exitCode=0
Oct 30 21:41:52.673310 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:52.673227    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b47v2" event={"ID":"1f3b6a23-d7ea-4ce4-b1e8-68133c4b4ebc","Type":"ContainerDied","Data":"9c3b30ec66ad103cdf19bbf59344440ab5f9f4677c7d202a77f05fb3a1191a22"}
Oct 30 21:41:52.678735 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:52.678704    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-sztmx" event={"ID":"238ff7d8-7da8-4cd8-9d45-deedf615211e","Type":"ContainerStarted","Data":"b3284e80939903abfb13e68157d48ff07f06aa27f537a280604d17a3ad756273"}
Oct 30 21:41:52.783087 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:52.783030    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-60.us-west-2.compute.internal" podStartSLOduration=6.783012096 podStartE2EDuration="6.783012096s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:41:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:52.781905645 +0000 UTC m=+7.592235668" watchObservedRunningTime="2026-10-30 21:41:52.783012096 +0000 UTC m=+7.593342120"
Oct 30 21:41:53.032902 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:53.032804    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-sztmx" podStartSLOduration=2.196527678 podStartE2EDuration="7.032786558s" totalImagesPullingTime="4.83625888s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:41:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:52.890008047 +0000 UTC m=+7.700338070" watchObservedRunningTime="2026-10-30 21:41:53.032786558 +0000 UTC m=+7.843116580"
Oct 30 21:41:54.518253 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:54.518228    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fe41c70b-7e2c-432f-8389-0282d6fdb332-original-pull-secret\") pod \"global-pull-secret-syncer-6wbw4\" (UID: \"fe41c70b-7e2c-432f-8389-0282d6fdb332\") " pod="kube-system/global-pull-secret-syncer-6wbw4"
Oct 30 21:41:54.518751 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:54.518297    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2205e8e6-f038-47fa-b393-0abe4d622e03-metrics-certs\") pod \"network-metrics-daemon-q6tjc\" (UID: \"2205e8e6-f038-47fa-b393-0abe4d622e03\") " pod="openshift-multus/network-metrics-daemon-q6tjc"
Oct 30 21:41:54.518751 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:41:54.518431    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:41:54.518751 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:41:54.518484    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2205e8e6-f038-47fa-b393-0abe4d622e03-metrics-certs podName:2205e8e6-f038-47fa-b393-0abe4d622e03 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:02.518467148 +0000 UTC m=+17.328797149 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2205e8e6-f038-47fa-b393-0abe4d622e03-metrics-certs") pod "network-metrics-daemon-q6tjc" (UID: "2205e8e6-f038-47fa-b393-0abe4d622e03") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:41:54.518751 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:41:54.518564    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 21:41:54.518751 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:41:54.518605    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe41c70b-7e2c-432f-8389-0282d6fdb332-original-pull-secret podName:fe41c70b-7e2c-432f-8389-0282d6fdb332 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:02.518592885 +0000 UTC m=+17.328922887 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fe41c70b-7e2c-432f-8389-0282d6fdb332-original-pull-secret") pod "global-pull-secret-syncer-6wbw4" (UID: "fe41c70b-7e2c-432f-8389-0282d6fdb332") : object "kube-system"/"original-pull-secret" not registered
Oct 30 21:41:54.611892 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:54.611361    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-979r9"
Oct 30 21:41:54.611892 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:41:54.611486    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-979r9" podUID="3784e680-5efe-4ffb-ad71-956d05b57dce"
Oct 30 21:41:54.611892 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:54.611568    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q6tjc"
Oct 30 21:41:54.611892 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:41:54.611675    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q6tjc" podUID="2205e8e6-f038-47fa-b393-0abe4d622e03"
Oct 30 21:41:54.611892 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:54.611721    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6wbw4"
Oct 30 21:41:54.611892 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:41:54.611813    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6wbw4" podUID="fe41c70b-7e2c-432f-8389-0282d6fdb332"
Oct 30 21:41:54.618809 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:54.618689    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-579v9\" (UniqueName: \"kubernetes.io/projected/3784e680-5efe-4ffb-ad71-956d05b57dce-kube-api-access-579v9\") pod \"network-check-target-979r9\" (UID: \"3784e680-5efe-4ffb-ad71-956d05b57dce\") " pod="openshift-network-diagnostics/network-check-target-979r9"
Oct 30 21:41:54.618944 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:41:54.618815    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 21:41:54.618944 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:41:54.618836    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 21:41:54.618944 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:41:54.618856    2580 projected.go:196] Error preparing data for projected volume kube-api-access-579v9 for pod openshift-network-diagnostics/network-check-target-979r9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:41:54.618944 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:41:54.618907    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3784e680-5efe-4ffb-ad71-956d05b57dce-kube-api-access-579v9 podName:3784e680-5efe-4ffb-ad71-956d05b57dce nodeName:}" failed. No retries permitted until 2026-10-30 21:42:02.618890983 +0000 UTC m=+17.429220995 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-579v9" (UniqueName: "kubernetes.io/projected/3784e680-5efe-4ffb-ad71-956d05b57dce-kube-api-access-579v9") pod "network-check-target-979r9" (UID: "3784e680-5efe-4ffb-ad71-956d05b57dce") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:41:54.686491 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:54.686458    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-x8gnj" event={"ID":"b227ad6a-9e59-415e-b7e6-f7c25e988abb","Type":"ContainerStarted","Data":"c371fb8f5332733648bf91b7f78d72eeeb940143b0d13dbe3aa6c7672d2854a5"}
Oct 30 21:41:54.689455 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:54.689063    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qk9dg" event={"ID":"4427f59d-8003-440f-9646-a46b176cc4a7","Type":"ContainerStarted","Data":"a12fbde44afb0e01e3b00cfdfa2ab2300671b1df19c3a2a54ba020fbcb735672"}
Oct 30 21:41:54.797180 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:54.797100    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-x8gnj" podStartSLOduration=3.442222737 podStartE2EDuration="8.79708092s" totalImagesPullingTime="5.354858183s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:41:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:54.796961071 +0000 UTC m=+9.607291094" watchObservedRunningTime="2026-10-30 21:41:54.79708092 +0000 UTC m=+9.607410945"
Oct 30 21:41:55.695176 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:55.695138    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-2wsrr" event={"ID":"71ddf455-cbf7-49a6-9773-30f7e4997a39","Type":"ContainerStarted","Data":"8a1fbc04c1c099907cedd656bfb540baa2e45f71e0566286b5eba0e2859c2922"}
Oct 30 21:41:55.807872 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:55.807812    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-2wsrr" podStartSLOduration=1.9590874889999998 podStartE2EDuration="9.807794799s" totalImagesPullingTime="7.84870731s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:41:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:55.807047247 +0000 UTC m=+10.617377268" watchObservedRunningTime="2026-10-30 21:41:55.807794799 +0000 UTC m=+10.618124823"
Oct 30 21:41:56.475353 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:56.475328    2580 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 21:41:56.604146 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:56.603991    2580 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T21:41:56.475350485Z","UUID":"bfd33644-ee34-4d0a-b46f-1f9dfc4eecf0","Handler":null,"Name":"","Endpoint":""}
Oct 30 21:41:56.605725 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:56.605705    2580 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 21:41:56.605832 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:56.605733    2580 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 21:41:56.611081 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:56.611053    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6wbw4"
Oct 30 21:41:56.611203 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:41:56.611167    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6wbw4" podUID="fe41c70b-7e2c-432f-8389-0282d6fdb332"
Oct 30 21:41:56.611270 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:56.611239    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-979r9"
Oct 30 21:41:56.611339 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:41:56.611318    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-979r9" podUID="3784e680-5efe-4ffb-ad71-956d05b57dce"
Oct 30 21:41:56.611394 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:56.611379    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q6tjc"
Oct 30 21:41:56.611491 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:41:56.611468    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q6tjc" podUID="2205e8e6-f038-47fa-b393-0abe4d622e03"
Oct 30 21:41:56.699898 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:56.699870    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qk9dg" event={"ID":"4427f59d-8003-440f-9646-a46b176cc4a7","Type":"ContainerStarted","Data":"7d032282cb9ff5ae53b00e09f5fc941d3eed0dc683f35fd309eb0d3f37069c46"}
Oct 30 21:41:56.702272 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:56.702244    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4nqkj" event={"ID":"a2e3b25e-b6dc-46ef-86e8-4e70f9dad200","Type":"ContainerStarted","Data":"58ee9a3af7f8eafc2303e19173dc2f72c86d41541588e7e5b741a38a85223962"}
Oct 30 21:41:56.821372 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:56.821311    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-4nqkj" podStartSLOduration=2.750077325 podStartE2EDuration="10.821294058s" totalImagesPullingTime="8.071216733s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:41:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:56.820681628 +0000 UTC m=+11.631011647" watchObservedRunningTime="2026-10-30 21:41:56.821294058 +0000 UTC m=+11.631624080"
Oct 30 21:41:56.941832 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:56.941517    2580 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-sztmx"
Oct 30 21:41:56.942142 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:56.942121    2580 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-sztmx"
Oct 30 21:41:57.708034 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:57.707694    2580 generic.go:356] "Generic (PLEG): container finished" podID="1f3b6a23-d7ea-4ce4-b1e8-68133c4b4ebc" containerID="5270a081e0c82cc2f7477dbab248e9aedcf934fbdcd6e5ae2af31fb686e18781" exitCode=0
Oct 30 21:41:57.708034 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:57.707795    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b47v2" event={"ID":"1f3b6a23-d7ea-4ce4-b1e8-68133c4b4ebc","Type":"ContainerDied","Data":"5270a081e0c82cc2f7477dbab248e9aedcf934fbdcd6e5ae2af31fb686e18781"}
Oct 30 21:41:57.711572 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:57.710796    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wbq4l" event={"ID":"0929b9f6-12ec-4f4d-8ba3-5cef9defba15","Type":"ContainerStarted","Data":"4428d3f0353eed5cc7c80d93a032c72dbf417e5031100003d6c41304fb94b2d2"}
Oct 30 21:41:57.711640 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:57.711577    2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-sztmx"
Oct 30 21:41:57.712203 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:57.712184    2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-sztmx"
Oct 30 21:41:58.141403 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:58.141352    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-wbq4l" podStartSLOduration=4.161495906 podStartE2EDuration="12.141332508s" totalImagesPullingTime="7.979836602s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:41:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:58.140688958 +0000 UTC m=+12.951018982" watchObservedRunningTime="2026-10-30 21:41:58.141332508 +0000 UTC m=+12.951662530"
Oct 30 21:41:58.611254 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:58.611173    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q6tjc"
Oct 30 21:41:58.611254 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:58.611204    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6wbw4"
Oct 30 21:41:58.611254 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:58.611221    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-979r9"
Oct 30 21:41:58.611462 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:41:58.611369    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q6tjc" podUID="2205e8e6-f038-47fa-b393-0abe4d622e03"
Oct 30 21:41:58.611462 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:41:58.611439    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-979r9" podUID="3784e680-5efe-4ffb-ad71-956d05b57dce"
Oct 30 21:41:58.611565 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:41:58.611549    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6wbw4" podUID="fe41c70b-7e2c-432f-8389-0282d6fdb332"
Oct 30 21:41:58.714523 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:58.714492    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-hskxk" event={"ID":"ed08bc51-70e6-4658-adb3-ed36f5176552","Type":"ContainerStarted","Data":"44d3d48e6dfdb33cc71e6cd184ee93536e7d08e1c5bfcfd3d699dc0506a4dc56"}
Oct 30 21:41:58.717513 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:58.717472    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qk9dg" event={"ID":"4427f59d-8003-440f-9646-a46b176cc4a7","Type":"ContainerStarted","Data":"984ad18f2663e1e9f58bdcb794694d185da0945daa7d76392db32ca196e4c48c"}
Oct 30 21:41:58.832302 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:58.832242    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-hskxk" podStartSLOduration=2.636572483 podStartE2EDuration="12.832228512s" totalImagesPullingTime="10.195656029s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:41:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:58.831864143 +0000 UTC m=+13.642194165" watchObservedRunningTime="2026-10-30 21:41:58.832228512 +0000 UTC m=+13.642558534"
Oct 30 21:41:58.960152 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:58.960065    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qk9dg" podStartSLOduration=3.218055451 podStartE2EDuration="12.960051425s" totalImagesPullingTime="9.741995974s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:41:46 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:58.95947452 +0000 UTC m=+13.769804541" watchObservedRunningTime="2026-10-30 21:41:58.960051425 +0000 UTC m=+13.770381447"
Oct 30 21:41:59.722024 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:59.721985    2580 generic.go:356] "Generic (PLEG): container finished" podID="1f3b6a23-d7ea-4ce4-b1e8-68133c4b4ebc" containerID="df74a38c1973f552b2f34ba504e4288306b7f3398f80f0068f4d63233c397d8f" exitCode=0
Oct 30 21:41:59.722734 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:41:59.722115    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b47v2" event={"ID":"1f3b6a23-d7ea-4ce4-b1e8-68133c4b4ebc","Type":"ContainerDied","Data":"df74a38c1973f552b2f34ba504e4288306b7f3398f80f0068f4d63233c397d8f"}
Oct 30 21:42:00.610735 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:00.610698    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6wbw4"
Oct 30 21:42:00.610908 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:00.610811    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q6tjc"
Oct 30 21:42:00.610960 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:00.610924    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-979r9"
Oct 30 21:42:00.610960 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:00.610942    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6wbw4" podUID="fe41c70b-7e2c-432f-8389-0282d6fdb332"
Oct 30 21:42:00.611032 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:00.610999    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-979r9" podUID="3784e680-5efe-4ffb-ad71-956d05b57dce"
Oct 30 21:42:00.611107 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:00.611081    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q6tjc" podUID="2205e8e6-f038-47fa-b393-0abe4d622e03"
Oct 30 21:42:01.728718 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:01.728400    2580 generic.go:356] "Generic (PLEG): container finished" podID="1f3b6a23-d7ea-4ce4-b1e8-68133c4b4ebc" containerID="f66cf20a6bc74068121170fff522486d3cfe02a8788157ebc69dea717abeb96e" exitCode=0
Oct 30 21:42:01.728718 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:01.728489    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b47v2" event={"ID":"1f3b6a23-d7ea-4ce4-b1e8-68133c4b4ebc","Type":"ContainerDied","Data":"f66cf20a6bc74068121170fff522486d3cfe02a8788157ebc69dea717abeb96e"}
Oct 30 21:42:02.580909 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:02.580883    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2205e8e6-f038-47fa-b393-0abe4d622e03-metrics-certs\") pod \"network-metrics-daemon-q6tjc\" (UID: \"2205e8e6-f038-47fa-b393-0abe4d622e03\") " pod="openshift-multus/network-metrics-daemon-q6tjc"
Oct 30 21:42:02.581014 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:02.580921    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fe41c70b-7e2c-432f-8389-0282d6fdb332-original-pull-secret\") pod \"global-pull-secret-syncer-6wbw4\" (UID: \"fe41c70b-7e2c-432f-8389-0282d6fdb332\") " pod="kube-system/global-pull-secret-syncer-6wbw4"
Oct 30 21:42:02.581057 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:02.581043    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:42:02.581086 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:02.581078    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 21:42:02.581116 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:02.581103    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2205e8e6-f038-47fa-b393-0abe4d622e03-metrics-certs podName:2205e8e6-f038-47fa-b393-0abe4d622e03 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:18.581087944 +0000 UTC m=+33.391417957 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2205e8e6-f038-47fa-b393-0abe4d622e03-metrics-certs") pod "network-metrics-daemon-q6tjc" (UID: "2205e8e6-f038-47fa-b393-0abe4d622e03") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:42:02.581157 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:02.581117    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe41c70b-7e2c-432f-8389-0282d6fdb332-original-pull-secret podName:fe41c70b-7e2c-432f-8389-0282d6fdb332 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:18.581111438 +0000 UTC m=+33.391441438 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fe41c70b-7e2c-432f-8389-0282d6fdb332-original-pull-secret") pod "global-pull-secret-syncer-6wbw4" (UID: "fe41c70b-7e2c-432f-8389-0282d6fdb332") : object "kube-system"/"original-pull-secret" not registered
Oct 30 21:42:02.610427 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:02.610403    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6wbw4"
Oct 30 21:42:02.610427 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:02.610422    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-979r9"
Oct 30 21:42:02.610546 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:02.610499    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q6tjc"
Oct 30 21:42:02.610546 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:02.610508    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-979r9" podUID="3784e680-5efe-4ffb-ad71-956d05b57dce"
Oct 30 21:42:02.610608 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:02.610568    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6wbw4" podUID="fe41c70b-7e2c-432f-8389-0282d6fdb332"
Oct 30 21:42:02.610672 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:02.610658    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q6tjc" podUID="2205e8e6-f038-47fa-b393-0abe4d622e03"
Oct 30 21:42:02.681499 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:02.681477    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-579v9\" (UniqueName: \"kubernetes.io/projected/3784e680-5efe-4ffb-ad71-956d05b57dce-kube-api-access-579v9\") pod \"network-check-target-979r9\" (UID: \"3784e680-5efe-4ffb-ad71-956d05b57dce\") " pod="openshift-network-diagnostics/network-check-target-979r9"
Oct 30 21:42:02.681623 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:02.681611    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 21:42:02.681678 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:02.681629    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 21:42:02.681678 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:02.681639    2580 projected.go:196] Error preparing data for projected volume kube-api-access-579v9 for pod openshift-network-diagnostics/network-check-target-979r9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:42:02.681746 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:02.681691    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3784e680-5efe-4ffb-ad71-956d05b57dce-kube-api-access-579v9 podName:3784e680-5efe-4ffb-ad71-956d05b57dce nodeName:}" failed. No retries permitted until 2026-10-30 21:42:18.681677633 +0000 UTC m=+33.492007641 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-579v9" (UniqueName: "kubernetes.io/projected/3784e680-5efe-4ffb-ad71-956d05b57dce-kube-api-access-579v9") pod "network-check-target-979r9" (UID: "3784e680-5efe-4ffb-ad71-956d05b57dce") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:42:02.733178 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:02.733029    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-22ldv" event={"ID":"538d5103-1c92-48f1-acc7-f90956848727","Type":"ContainerStarted","Data":"caec0dda6447012ba2b59ce61f9ec3e276ef096b312f705f14b06f55463d1129"}
Oct 30 21:42:02.733554 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:02.733216    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-22ldv" event={"ID":"538d5103-1c92-48f1-acc7-f90956848727","Type":"ContainerStarted","Data":"c68acab3833217d613c53eddca66b199d428bb234beae2d3b7de76e891d8871f"}
Oct 30 21:42:02.733554 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:02.733246    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-22ldv" event={"ID":"538d5103-1c92-48f1-acc7-f90956848727","Type":"ContainerStarted","Data":"f9650436fbea220784a14aa7a316faad252f513a98631bd299a03bbe2f838493"}
Oct 30 21:42:02.733554 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:02.733274    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-22ldv" event={"ID":"538d5103-1c92-48f1-acc7-f90956848727","Type":"ContainerStarted","Data":"b16524c3427f1741c162e84550004f361cc4d44ac49bb5529b20291862ad71a5"}
Oct 30 21:42:03.738399 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:03.738361    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-22ldv" event={"ID":"538d5103-1c92-48f1-acc7-f90956848727","Type":"ContainerStarted","Data":"9d8e6d5da066b3a5cd089467e463c35674827e8627f0b1922a3e1c0f8f536ba0"}
Oct 30 21:42:03.738871 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:03.738448    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-22ldv" event={"ID":"538d5103-1c92-48f1-acc7-f90956848727","Type":"ContainerStarted","Data":"a699a2d6b94b524087c3d2aebf9c7df38ee8ce5f542424f24dcbfcd8af9d02cc"}
Oct 30 21:42:04.611242 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:04.611177    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6wbw4"
Oct 30 21:42:04.611242 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:04.611216    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-979r9"
Oct 30 21:42:04.611431 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:04.611287    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q6tjc"
Oct 30 21:42:04.611431 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:04.611309    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6wbw4" podUID="fe41c70b-7e2c-432f-8389-0282d6fdb332"
Oct 30 21:42:04.611431 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:04.611360    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q6tjc" podUID="2205e8e6-f038-47fa-b393-0abe4d622e03"
Oct 30 21:42:04.611431 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:04.611411    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-979r9" podUID="3784e680-5efe-4ffb-ad71-956d05b57dce"
Oct 30 21:42:04.742875 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:04.742846    2580 generic.go:356] "Generic (PLEG): container finished" podID="1f3b6a23-d7ea-4ce4-b1e8-68133c4b4ebc" containerID="84b653b15569d0c3ddcec103af4165ac52ce183d9e0ef60757af9badc5079ff7" exitCode=0
Oct 30 21:42:04.743280 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:04.742895    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b47v2" event={"ID":"1f3b6a23-d7ea-4ce4-b1e8-68133c4b4ebc","Type":"ContainerDied","Data":"84b653b15569d0c3ddcec103af4165ac52ce183d9e0ef60757af9badc5079ff7"}
Oct 30 21:42:05.747401 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:05.747372    2580 generic.go:356] "Generic (PLEG): container finished" podID="1f3b6a23-d7ea-4ce4-b1e8-68133c4b4ebc" containerID="4d2c6ac7e5209c068013758686fb1e2f2292f9fda8333cf1f97cf491ae8201b6" exitCode=0
Oct 30 21:42:05.747898 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:05.747415    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b47v2" event={"ID":"1f3b6a23-d7ea-4ce4-b1e8-68133c4b4ebc","Type":"ContainerDied","Data":"4d2c6ac7e5209c068013758686fb1e2f2292f9fda8333cf1f97cf491ae8201b6"}
Oct 30 21:42:05.750620 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:05.750599    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-22ldv" event={"ID":"538d5103-1c92-48f1-acc7-f90956848727","Type":"ContainerStarted","Data":"5715c40a24ee3e611b3f592515b308b9118a59f6c8397afb3ea678c34255b608"}
Oct 30 21:42:06.611278 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:06.611247    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-979r9"
Oct 30 21:42:06.611278 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:06.611274    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6wbw4"
Oct 30 21:42:06.611451 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:06.611344    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-979r9" podUID="3784e680-5efe-4ffb-ad71-956d05b57dce"
Oct 30 21:42:06.611451 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:06.611379    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q6tjc"
Oct 30 21:42:06.611514 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:06.611456    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q6tjc" podUID="2205e8e6-f038-47fa-b393-0abe4d622e03"
Oct 30 21:42:06.611514 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:06.611497    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6wbw4" podUID="fe41c70b-7e2c-432f-8389-0282d6fdb332"
Oct 30 21:42:06.755851 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:06.755823    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b47v2" event={"ID":"1f3b6a23-d7ea-4ce4-b1e8-68133c4b4ebc","Type":"ContainerStarted","Data":"e68d178b3e883d33a6a755f05e94f1763bc570118ca42aa8dfd1813e5fd678a5"}
Oct 30 21:42:07.761127 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:07.760891    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-22ldv" event={"ID":"538d5103-1c92-48f1-acc7-f90956848727","Type":"ContainerStarted","Data":"b89f9593d6638a0096af303bef5534007a7ba2459c02b8db608db1342c2d4ee7"}
Oct 30 21:42:08.027795 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:08.027667    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-22ldv" podStartSLOduration=7.602315712 podStartE2EDuration="22.027653014s" totalImagesPullingTime="14.425337302s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:41:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:07.980131065 +0000 UTC m=+22.790461087" watchObservedRunningTime="2026-10-30 21:42:08.027653014 +0000 UTC m=+22.837983039"
Oct 30 21:42:08.028014 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:08.027986    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-b47v2" podStartSLOduration=8.936597958 podStartE2EDuration="22.027977714s" totalImagesPullingTime="13.091379756s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:41:46 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:06.893280295 +0000 UTC m=+21.703610317" watchObservedRunningTime="2026-10-30 21:42:08.027977714 +0000 UTC m=+22.838307735"
Oct 30 21:42:08.610661 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:08.610630    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6wbw4"
Oct 30 21:42:08.610835 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:08.610745    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-979r9"
Oct 30 21:42:08.610835 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:08.610755    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6wbw4" podUID="fe41c70b-7e2c-432f-8389-0282d6fdb332"
Oct 30 21:42:08.610905 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:08.610833    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-979r9" podUID="3784e680-5efe-4ffb-ad71-956d05b57dce"
Oct 30 21:42:08.610905 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:08.610888    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q6tjc"
Oct 30 21:42:08.610967 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:08.610950    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q6tjc" podUID="2205e8e6-f038-47fa-b393-0abe4d622e03"
Oct 30 21:42:08.763234 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:08.763205    2580 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 21:42:08.763635 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:08.763569    2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-22ldv"
Oct 30 21:42:08.763702 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:08.763645    2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-22ldv"
Oct 30 21:42:08.779112 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:08.779092    2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-22ldv"
Oct 30 21:42:08.779230 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:08.779204    2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-22ldv"
Oct 30 21:42:09.765675 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:09.765646    2580 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 21:42:09.794182 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:09.794155    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-q6tjc"]
Oct 30 21:42:09.794332 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:09.794258    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q6tjc"
Oct 30 21:42:09.794380 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:09.794362    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q6tjc" podUID="2205e8e6-f038-47fa-b393-0abe4d622e03"
Oct 30 21:42:09.794658 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:09.794641    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6wbw4"]
Oct 30 21:42:09.794723 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:09.794713    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6wbw4"
Oct 30 21:42:09.794799 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:09.794784    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6wbw4" podUID="fe41c70b-7e2c-432f-8389-0282d6fdb332"
Oct 30 21:42:09.804825 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:09.804806    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-979r9"]
Oct 30 21:42:09.804921 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:09.804873    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-979r9"
Oct 30 21:42:09.804959 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:09.804933    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-979r9" podUID="3784e680-5efe-4ffb-ad71-956d05b57dce"
Oct 30 21:42:10.767872 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:10.767844    2580 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 21:42:11.611426 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:11.611389    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q6tjc"
Oct 30 21:42:11.611599 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:11.611494    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-979r9"
Oct 30 21:42:11.611599 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:11.611501    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q6tjc" podUID="2205e8e6-f038-47fa-b393-0abe4d622e03"
Oct 30 21:42:11.611599 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:11.611591    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-979r9" podUID="3784e680-5efe-4ffb-ad71-956d05b57dce"
Oct 30 21:42:11.611714 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:11.611613    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6wbw4"
Oct 30 21:42:11.611714 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:11.611659    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6wbw4" podUID="fe41c70b-7e2c-432f-8389-0282d6fdb332"
Oct 30 21:42:12.039927 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:12.039856    2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-22ldv"
Oct 30 21:42:12.040350 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:12.040100    2580 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 21:42:12.050604 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:12.050574    2580 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-22ldv" podUID="538d5103-1c92-48f1-acc7-f90956848727" containerName="ovnkube-controller" probeResult="failure" output=""
Oct 30 21:42:12.058658 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:12.058633    2580 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-22ldv" podUID="538d5103-1c92-48f1-acc7-f90956848727" containerName="ovnkube-controller" probeResult="failure" output=""
Oct 30 21:42:13.610959 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:13.610933    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-979r9"
Oct 30 21:42:13.610959 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:13.610933    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q6tjc"
Oct 30 21:42:13.611336 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:13.611080    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-979r9" podUID="3784e680-5efe-4ffb-ad71-956d05b57dce"
Oct 30 21:42:13.611336 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:13.611120    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6wbw4"
Oct 30 21:42:13.611336 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:13.611203    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q6tjc" podUID="2205e8e6-f038-47fa-b393-0abe4d622e03"
Oct 30 21:42:13.611336 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:13.611263    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6wbw4" podUID="fe41c70b-7e2c-432f-8389-0282d6fdb332"
Oct 30 21:42:14.985197 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:14.985174    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-60.us-west-2.compute.internal" event="NodeReady"
Oct 30 21:42:14.985549 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:14.985291    2580 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 30 21:42:15.098156 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.097976    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-kz6dl"]
Oct 30 21:42:15.103977 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.103963    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-bgnkz"]
Oct 30 21:42:15.104108 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.104090    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-kz6dl"
Oct 30 21:42:15.111100 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.111081    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-ns759"]
Oct 30 21:42:15.111177 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.111156    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-bgnkz"
Oct 30 21:42:15.117386 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.117369    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-cklcw"]
Oct 30 21:42:15.117508 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.117493    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ns759"
Oct 30 21:42:15.126059 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.126040    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-cklcw"
Oct 30 21:42:15.147979 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.147923    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-6c7f96b45-4wj99"]
Oct 30 21:42:15.159160 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.159138    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-s6427"]
Oct 30 21:42:15.159292 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.159275    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6c7f96b45-4wj99"
Oct 30 21:42:15.169514 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.169496    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-d44c8cbb7-lmw97"]
Oct 30 21:42:15.169635 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.169594    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-s6427"
Oct 30 21:42:15.180166 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.180149    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-mgxcj"]
Oct 30 21:42:15.180269 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.180255    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-d44c8cbb7-lmw97"
Oct 30 21:42:15.194360 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.194339    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-l9b9d"]
Oct 30 21:42:15.194457 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.194435    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-mgxcj"
Oct 30 21:42:15.194915 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.194892    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Oct 30 21:42:15.194915 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.194894    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Oct 30 21:42:15.197078 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.197039    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-5m5gq\""
Oct 30 21:42:15.206891 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.206874    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-sclrt"]
Oct 30 21:42:15.207003 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.206988    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-l9b9d"
Oct 30 21:42:15.217798 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.217780    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-tlwms"]
Oct 30 21:42:15.217898 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.217876    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-sclrt"
Oct 30 21:42:15.235775 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.235756    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-vqwlt"]
Oct 30 21:42:15.235873 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.235861    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-tlwms"
Oct 30 21:42:15.249958 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.249941    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-b6cdbd45d-bdftd"]
Oct 30 21:42:15.250059 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.250044    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vqwlt"
Oct 30 21:42:15.257074 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.257057    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 30 21:42:15.257383 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.257358    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 30 21:42:15.257383 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.257370    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Oct 30 21:42:15.257520 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.257398    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 21:42:15.257636 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.257619    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 21:42:15.257765 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.257624    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Oct 30 21:42:15.257765 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.257656    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Oct 30 21:42:15.257878 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.257658    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Oct 30 21:42:15.257878 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.257668    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 30 21:42:15.258139 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.258126    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Oct 30 21:42:15.258193 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.258185    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 21:42:15.258226 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.258185    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Oct 30 21:42:15.258226 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.258223    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Oct 30 21:42:15.260274 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.260253    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-rmpxf"]
Oct 30 21:42:15.260388 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.260374    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-b6cdbd45d-bdftd"
Oct 30 21:42:15.270495 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.270473    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-j5xsr"]
Oct 30 21:42:15.271600 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.271509    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-rmpxf"
Oct 30 21:42:15.274707 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.274682    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/613e7c8a-e2c7-424a-93e0-7fa86d18a8a9-image-registry-private-configuration\") pod \"image-registry-d44c8cbb7-lmw97\" (UID: \"613e7c8a-e2c7-424a-93e0-7fa86d18a8a9\") " pod="openshift-image-registry/image-registry-d44c8cbb7-lmw97"
Oct 30 21:42:15.274924 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.274722    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fcz7\" (UniqueName: \"kubernetes.io/projected/102b8049-7349-4e2b-974a-44c901eebcf3-kube-api-access-5fcz7\") pod \"cluster-monitoring-operator-884576c57-ns759\" (UID: \"102b8049-7349-4e2b-974a-44c901eebcf3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ns759"
Oct 30 21:42:15.274924 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.274753    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/613e7c8a-e2c7-424a-93e0-7fa86d18a8a9-ca-trust-extracted\") pod \"image-registry-d44c8cbb7-lmw97\" (UID: \"613e7c8a-e2c7-424a-93e0-7fa86d18a8a9\") " pod="openshift-image-registry/image-registry-d44c8cbb7-lmw97"
Oct 30 21:42:15.274924 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.274785    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d-stats-auth\") pod \"router-default-6c7f96b45-4wj99\" (UID: \"9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d\") " pod="openshift-ingress/router-default-6c7f96b45-4wj99"
Oct 30 21:42:15.274924 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.274810    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/798f248f-d429-4f56-8752-e157dd94206b-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-mgxcj\" (UID: \"798f248f-d429-4f56-8752-e157dd94206b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-mgxcj"
Oct 30 21:42:15.274924 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.274835    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdtc5\" (UniqueName: \"kubernetes.io/projected/3f97ee34-5dfc-454e-ba5c-afd939f3bd25-kube-api-access-cdtc5\") pod \"kas-connection-checker-685895c7c9-bgnkz\" (UID: \"3f97ee34-5dfc-454e-ba5c-afd939f3bd25\") " pod="kube-system/kas-connection-checker-685895c7c9-bgnkz"
Oct 30 21:42:15.274924 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.274864    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/613e7c8a-e2c7-424a-93e0-7fa86d18a8a9-trusted-ca\") pod \"image-registry-d44c8cbb7-lmw97\" (UID: \"613e7c8a-e2c7-424a-93e0-7fa86d18a8a9\") " pod="openshift-image-registry/image-registry-d44c8cbb7-lmw97"
Oct 30 21:42:15.274924 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.274885    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/613e7c8a-e2c7-424a-93e0-7fa86d18a8a9-installation-pull-secrets\") pod \"image-registry-d44c8cbb7-lmw97\" (UID: \"613e7c8a-e2c7-424a-93e0-7fa86d18a8a9\") " pod="openshift-image-registry/image-registry-d44c8cbb7-lmw97"
Oct 30 21:42:15.274924 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.274910    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d-metrics-certs\") pod \"router-default-6c7f96b45-4wj99\" (UID: \"9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d\") " pod="openshift-ingress/router-default-6c7f96b45-4wj99"
Oct 30 21:42:15.275348 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.274932    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/613e7c8a-e2c7-424a-93e0-7fa86d18a8a9-registry-certificates\") pod \"image-registry-d44c8cbb7-lmw97\" (UID: \"613e7c8a-e2c7-424a-93e0-7fa86d18a8a9\") " pod="openshift-image-registry/image-registry-d44c8cbb7-lmw97"
Oct 30 21:42:15.275348 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.274957    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/102b8049-7349-4e2b-974a-44c901eebcf3-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-ns759\" (UID: \"102b8049-7349-4e2b-974a-44c901eebcf3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ns759"
Oct 30 21:42:15.275348 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.274982    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/102b8049-7349-4e2b-974a-44c901eebcf3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-ns759\" (UID: \"102b8049-7349-4e2b-974a-44c901eebcf3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ns759"
Oct 30 21:42:15.275348 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.275013    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xk5zb\" (UniqueName: \"kubernetes.io/projected/c798fd57-e0be-4b61-860b-0b833aca7784-kube-api-access-xk5zb\") pod \"kas-connection-checker-685895c7c9-kz6dl\" (UID: \"c798fd57-e0be-4b61-860b-0b833aca7784\") " pod="kube-system/kas-connection-checker-685895c7c9-kz6dl"
Oct 30 21:42:15.275348 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.275056    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4tb7\" (UniqueName: \"kubernetes.io/projected/9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d-kube-api-access-n4tb7\") pod \"router-default-6c7f96b45-4wj99\" (UID: \"9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d\") " pod="openshift-ingress/router-default-6c7f96b45-4wj99"
Oct 30 21:42:15.275348 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.275088    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/798f248f-d429-4f56-8752-e157dd94206b-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-mgxcj\" (UID: \"798f248f-d429-4f56-8752-e157dd94206b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-mgxcj"
Oct 30 21:42:15.275348 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.275136    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/613e7c8a-e2c7-424a-93e0-7fa86d18a8a9-bound-sa-token\") pod \"image-registry-d44c8cbb7-lmw97\" (UID: \"613e7c8a-e2c7-424a-93e0-7fa86d18a8a9\") " pod="openshift-image-registry/image-registry-d44c8cbb7-lmw97"
Oct 30 21:42:15.275348 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.275159    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-424gm\" (UniqueName: \"kubernetes.io/projected/a32aacdc-53be-41cf-839e-435d43361890-kube-api-access-424gm\") pod \"kas-connection-checker-685895c7c9-cklcw\" (UID: \"a32aacdc-53be-41cf-839e-435d43361890\") " pod="kube-system/kas-connection-checker-685895c7c9-cklcw"
Oct 30 21:42:15.275348 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.275183    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhbrj\" (UniqueName: \"kubernetes.io/projected/d700d1eb-fbf4-45df-9b57-eca08d87ebe2-kube-api-access-nhbrj\") pod \"volume-data-source-validator-58c9f5f46d-s6427\" (UID: \"d700d1eb-fbf4-45df-9b57-eca08d87ebe2\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-s6427"
Oct 30 21:42:15.275348 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.275208    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hw226\" (UniqueName: \"kubernetes.io/projected/613e7c8a-e2c7-424a-93e0-7fa86d18a8a9-kube-api-access-hw226\") pod \"image-registry-d44c8cbb7-lmw97\" (UID: \"613e7c8a-e2c7-424a-93e0-7fa86d18a8a9\") " pod="openshift-image-registry/image-registry-d44c8cbb7-lmw97"
Oct 30 21:42:15.275348 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.275235    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d-service-ca-bundle\") pod \"router-default-6c7f96b45-4wj99\" (UID: \"9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d\") " pod="openshift-ingress/router-default-6c7f96b45-4wj99"
Oct 30 21:42:15.275348 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.275259    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/613e7c8a-e2c7-424a-93e0-7fa86d18a8a9-registry-tls\") pod \"image-registry-d44c8cbb7-lmw97\" (UID: \"613e7c8a-e2c7-424a-93e0-7fa86d18a8a9\") " pod="openshift-image-registry/image-registry-d44c8cbb7-lmw97"
Oct 30 21:42:15.275348 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.275289    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d-default-certificate\") pod \"router-default-6c7f96b45-4wj99\" (UID: \"9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d\") " pod="openshift-ingress/router-default-6c7f96b45-4wj99"
Oct 30 21:42:15.275348 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.275314    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8w7g2\" (UniqueName: \"kubernetes.io/projected/798f248f-d429-4f56-8752-e157dd94206b-kube-api-access-8w7g2\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-mgxcj\" (UID: \"798f248f-d429-4f56-8752-e157dd94206b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-mgxcj"
Oct 30 21:42:15.282252 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.282230    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-6pttx"]
Oct 30 21:42:15.282361 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.282348    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-j5xsr"
Oct 30 21:42:15.292446 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.288954    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-pzpgw"]
Oct 30 21:42:15.292446 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.289176    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6pttx"
Oct 30 21:42:15.297289 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.296960    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-kz6dl"]
Oct 30 21:42:15.297289 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.297000    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-bgnkz"]
Oct 30 21:42:15.297289 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.297023    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-cklcw"]
Oct 30 21:42:15.297289 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.297043    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-ns759"]
Oct 30 21:42:15.297289 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.297065    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-s6427"]
Oct 30 21:42:15.297289 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.297087    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-mgxcj"]
Oct 30 21:42:15.297289 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.297102    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-pzpgw"
Oct 30 21:42:15.297689 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.297106    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-l9b9d"]
Oct 30 21:42:15.297689 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.297492    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-vqwlt"]
Oct 30 21:42:15.304417 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.304401    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 21:42:15.306047 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.306028    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 30 21:42:15.306392 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.306372    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 30 21:42:15.308694 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.308678    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-rmpxf"]
Oct 30 21:42:15.309399 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.309384    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-j5xsr"]
Oct 30 21:42:15.310036 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.310018    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-sclrt"]
Oct 30 21:42:15.310314 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.310297    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 30 21:42:15.310804 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.310782    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 30 21:42:15.310890 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.310845    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Oct 30 21:42:15.311169 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.311151    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6c7f96b45-4wj99"]
Oct 30 21:42:15.312219 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.312201    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Oct 30 21:42:15.312219 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.312217    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Oct 30 21:42:15.315162 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.315142    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Oct 30 21:42:15.315260 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.315183    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-k8d6g\""
Oct 30 21:42:15.315260 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.315195    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Oct 30 21:42:15.315260 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.315207    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-mxjzl\""
Oct 30 21:42:15.315260 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.315236    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-rzsz9\""
Oct 30 21:42:15.315518 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.315496    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Oct 30 21:42:15.316230 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.316212    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-rqz2k\""
Oct 30 21:42:15.316319 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.316241    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Oct 30 21:42:15.317214 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.316562    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-ngcxp\""
Oct 30 21:42:15.317214 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.316566    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 30 21:42:15.317214 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.316677    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-4rd6j\""
Oct 30 21:42:15.317792 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.317774    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-b6cdbd45d-bdftd"]
Oct 30 21:42:15.319466 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.319446    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-pzpgw"]
Oct 30 21:42:15.328665 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.328646    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-tlwms"]
Oct 30 21:42:15.329301 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.329281    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-d44c8cbb7-lmw97"]
Oct 30 21:42:15.346056 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.346038    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-6pttx"]
Oct 30 21:42:15.357792 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.357775    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Oct 30 21:42:15.357997 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.357983    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Oct 30 21:42:15.358308 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.358297    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Oct 30 21:42:15.358452 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.358432    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Oct 30 21:42:15.358587 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.358476    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 21:42:15.358587 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.358484    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 21:42:15.358587 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.358498    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 21:42:15.358587 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.358518    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Oct 30 21:42:15.358768 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.358755    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Oct 30 21:42:15.358914 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.358898    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Oct 30 21:42:15.358982 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.358929    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Oct 30 21:42:15.358982 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.358954    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Oct 30 21:42:15.359067 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.359044    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 21:42:15.359255 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.359243    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Oct 30 21:42:15.359333 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.359283    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 21:42:15.359638 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.359523    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Oct 30 21:42:15.373055 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.373034    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Oct 30 21:42:15.373156 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.373137    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Oct 30 21:42:15.373156 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.373152    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Oct 30 21:42:15.373266 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.373199    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-g4k97\""
Oct 30 21:42:15.373266 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.373235    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-zqhcb\""
Oct 30 21:42:15.373266 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.373258    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-j86j7\""
Oct 30 21:42:15.373266 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.373266    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 21:42:15.373417 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.373266    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 21:42:15.373417 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.373296    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-kphrd\""
Oct 30 21:42:15.373417 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.373376    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-cnbh6\""
Oct 30 21:42:15.373417 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.373384    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-ssgdc\""
Oct 30 21:42:15.375570 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.375553    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/c69312da-8be3-4931-8a81-a8ff3f86d5e6-snapshots\") pod \"insights-operator-7c4986d449-l9b9d\" (UID: \"c69312da-8be3-4931-8a81-a8ff3f86d5e6\") " pod="openshift-insights/insights-operator-7c4986d449-l9b9d"
Oct 30 21:42:15.375645 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.375586    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l75p7\" (UniqueName: \"kubernetes.io/projected/28210ad9-18f0-4ecf-9bfc-b256440f0ce8-kube-api-access-l75p7\") pod \"ingress-canary-pzpgw\" (UID: \"28210ad9-18f0-4ecf-9bfc-b256440f0ce8\") " pod="openshift-ingress-canary/ingress-canary-pzpgw"
Oct 30 21:42:15.375645 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.375620    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/613e7c8a-e2c7-424a-93e0-7fa86d18a8a9-image-registry-private-configuration\") pod \"image-registry-d44c8cbb7-lmw97\" (UID: \"613e7c8a-e2c7-424a-93e0-7fa86d18a8a9\") " pod="openshift-image-registry/image-registry-d44c8cbb7-lmw97"
Oct 30 21:42:15.375713 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.375659    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cad67bc5-c8d2-497c-ba1b-cdbae5822cfa-bound-sa-token\") pod \"image-registry-b6cdbd45d-bdftd\" (UID: \"cad67bc5-c8d2-497c-ba1b-cdbae5822cfa\") " pod="openshift-image-registry/image-registry-b6cdbd45d-bdftd"
Oct 30 21:42:15.375773 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.375757    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/613e7c8a-e2c7-424a-93e0-7fa86d18a8a9-ca-trust-extracted\") pod \"image-registry-d44c8cbb7-lmw97\" (UID: \"613e7c8a-e2c7-424a-93e0-7fa86d18a8a9\") " pod="openshift-image-registry/image-registry-d44c8cbb7-lmw97"
Oct 30 21:42:15.375826 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.375810    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kr7ww\" (UniqueName: \"kubernetes.io/projected/20f55ce3-5b04-4916-b529-0dd596aba68f-kube-api-access-kr7ww\") pod \"network-check-source-6f78d59f99-rmpxf\" (UID: \"20f55ce3-5b04-4916-b529-0dd596aba68f\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-rmpxf"
Oct 30 21:42:15.375876 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.375847    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/0bb51175-fa5e-4472-8d83-106bca721585-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-tlwms\" (UID: \"0bb51175-fa5e-4472-8d83-106bca721585\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-tlwms"
Oct 30 21:42:15.375912 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.375890    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdtc5\" (UniqueName: \"kubernetes.io/projected/3f97ee34-5dfc-454e-ba5c-afd939f3bd25-kube-api-access-cdtc5\") pod \"kas-connection-checker-685895c7c9-bgnkz\" (UID: \"3f97ee34-5dfc-454e-ba5c-afd939f3bd25\") " pod="kube-system/kas-connection-checker-685895c7c9-bgnkz"
Oct 30 21:42:15.375987 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.375973    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/613e7c8a-e2c7-424a-93e0-7fa86d18a8a9-installation-pull-secrets\") pod \"image-registry-d44c8cbb7-lmw97\" (UID: \"613e7c8a-e2c7-424a-93e0-7fa86d18a8a9\") " pod="openshift-image-registry/image-registry-d44c8cbb7-lmw97"
Oct 30 21:42:15.376079 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.376057    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c69312da-8be3-4931-8a81-a8ff3f86d5e6-tmp\") pod \"insights-operator-7c4986d449-l9b9d\" (UID: \"c69312da-8be3-4931-8a81-a8ff3f86d5e6\") " pod="openshift-insights/insights-operator-7c4986d449-l9b9d"
Oct 30 21:42:15.376170 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.376129    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5e74071e-0462-42b5-a05c-d90bb638478a-trusted-ca\") pod \"console-operator-7cd56b8dbc-sclrt\" (UID: \"5e74071e-0462-42b5-a05c-d90bb638478a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-sclrt"
Oct 30 21:42:15.376170 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.376152    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/613e7c8a-e2c7-424a-93e0-7fa86d18a8a9-ca-trust-extracted\") pod \"image-registry-d44c8cbb7-lmw97\" (UID: \"613e7c8a-e2c7-424a-93e0-7fa86d18a8a9\") " pod="openshift-image-registry/image-registry-d44c8cbb7-lmw97"
Oct 30 21:42:15.376247 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.376188    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d-metrics-certs\") pod \"router-default-6c7f96b45-4wj99\" (UID: \"9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d\") " pod="openshift-ingress/router-default-6c7f96b45-4wj99"
Oct 30 21:42:15.376247 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.376221    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0bb51175-fa5e-4472-8d83-106bca721585-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-tlwms\" (UID: \"0bb51175-fa5e-4472-8d83-106bca721585\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-tlwms"
Oct 30 21:42:15.376315 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.376249    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdbmm\" (UniqueName: \"kubernetes.io/projected/0bb51175-fa5e-4472-8d83-106bca721585-kube-api-access-pdbmm\") pod \"cluster-samples-operator-6ddb6c8b94-tlwms\" (UID: \"0bb51175-fa5e-4472-8d83-106bca721585\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-tlwms"
Oct 30 21:42:15.376315 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.376296    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/798f248f-d429-4f56-8752-e157dd94206b-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-mgxcj\" (UID: \"798f248f-d429-4f56-8752-e157dd94206b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-mgxcj"
Oct 30 21:42:15.376423 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:15.376316    2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 21:42:15.376423 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.376321    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/28210ad9-18f0-4ecf-9bfc-b256440f0ce8-cert\") pod \"ingress-canary-pzpgw\" (UID: \"28210ad9-18f0-4ecf-9bfc-b256440f0ce8\") " pod="openshift-ingress-canary/ingress-canary-pzpgw"
Oct 30 21:42:15.376423 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.376355    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srwt7\" (UniqueName: \"kubernetes.io/projected/0126a0a0-a1c8-4db3-a6af-629360e247f4-kube-api-access-srwt7\") pod \"service-ca-operator-544d749c8c-vqwlt\" (UID: \"0126a0a0-a1c8-4db3-a6af-629360e247f4\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vqwlt"
Oct 30 21:42:15.376423 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.376381    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cad67bc5-c8d2-497c-ba1b-cdbae5822cfa-registry-certificates\") pod \"image-registry-b6cdbd45d-bdftd\" (UID: \"cad67bc5-c8d2-497c-ba1b-cdbae5822cfa\") " pod="openshift-image-registry/image-registry-b6cdbd45d-bdftd"
Oct 30 21:42:15.376423 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.376399    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hw226\" (UniqueName: \"kubernetes.io/projected/613e7c8a-e2c7-424a-93e0-7fa86d18a8a9-kube-api-access-hw226\") pod \"image-registry-d44c8cbb7-lmw97\" (UID: \"613e7c8a-e2c7-424a-93e0-7fa86d18a8a9\") " pod="openshift-image-registry/image-registry-d44c8cbb7-lmw97"
Oct 30 21:42:15.376647 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.376433    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/84d9b682-3241-40af-87d5-f87d54ce3697-config-volume\") pod \"dns-default-6pttx\" (UID: \"84d9b682-3241-40af-87d5-f87d54ce3697\") " pod="openshift-dns/dns-default-6pttx"
Oct 30 21:42:15.376647 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:15.376478    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d-metrics-certs podName:9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d nodeName:}" failed. No retries permitted until 2026-10-30 21:42:15.876459682 +0000 UTC m=+30.686789703 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d-metrics-certs") pod "router-default-6c7f96b45-4wj99" (UID: "9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d") : secret "router-metrics-certs-default" not found
Oct 30 21:42:15.376647 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.376574    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d-default-certificate\") pod \"router-default-6c7f96b45-4wj99\" (UID: \"9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d\") " pod="openshift-ingress/router-default-6c7f96b45-4wj99"
Oct 30 21:42:15.376647 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.376617    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8w7g2\" (UniqueName: \"kubernetes.io/projected/798f248f-d429-4f56-8752-e157dd94206b-kube-api-access-8w7g2\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-mgxcj\" (UID: \"798f248f-d429-4f56-8752-e157dd94206b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-mgxcj"
Oct 30 21:42:15.376647 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.376643    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0126a0a0-a1c8-4db3-a6af-629360e247f4-serving-cert\") pod \"service-ca-operator-544d749c8c-vqwlt\" (UID: \"0126a0a0-a1c8-4db3-a6af-629360e247f4\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vqwlt"
Oct 30 21:42:15.376840 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.376668    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fcz7\" (UniqueName: \"kubernetes.io/projected/102b8049-7349-4e2b-974a-44c901eebcf3-kube-api-access-5fcz7\") pod \"cluster-monitoring-operator-884576c57-ns759\" (UID: \"102b8049-7349-4e2b-974a-44c901eebcf3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ns759"
Oct 30 21:42:15.376840 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.376690    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cad67bc5-c8d2-497c-ba1b-cdbae5822cfa-registry-tls\") pod \"image-registry-b6cdbd45d-bdftd\" (UID: \"cad67bc5-c8d2-497c-ba1b-cdbae5822cfa\") " pod="openshift-image-registry/image-registry-b6cdbd45d-bdftd"
Oct 30 21:42:15.376840 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.376726    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cad67bc5-c8d2-497c-ba1b-cdbae5822cfa-ca-trust-extracted\") pod \"image-registry-b6cdbd45d-bdftd\" (UID: \"cad67bc5-c8d2-497c-ba1b-cdbae5822cfa\") " pod="openshift-image-registry/image-registry-b6cdbd45d-bdftd"
Oct 30 21:42:15.376840 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.376771    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cad67bc5-c8d2-497c-ba1b-cdbae5822cfa-installation-pull-secrets\") pod \"image-registry-b6cdbd45d-bdftd\" (UID: \"cad67bc5-c8d2-497c-ba1b-cdbae5822cfa\") " pod="openshift-image-registry/image-registry-b6cdbd45d-bdftd"
Oct 30 21:42:15.376840 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.376795    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9dpv\" (UniqueName: \"kubernetes.io/projected/cad67bc5-c8d2-497c-ba1b-cdbae5822cfa-kube-api-access-m9dpv\") pod \"image-registry-b6cdbd45d-bdftd\" (UID: \"cad67bc5-c8d2-497c-ba1b-cdbae5822cfa\") " pod="openshift-image-registry/image-registry-b6cdbd45d-bdftd"
Oct 30 21:42:15.377042 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.376909    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f648e933-f035-486a-88d5-1b185c89782e-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-j5xsr\" (UID: \"f648e933-f035-486a-88d5-1b185c89782e\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-j5xsr"
Oct 30 21:42:15.377042 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.376945    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d-stats-auth\") pod \"router-default-6c7f96b45-4wj99\" (UID: \"9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d\") " pod="openshift-ingress/router-default-6c7f96b45-4wj99"
Oct 30 21:42:15.377042 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.376970    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/798f248f-d429-4f56-8752-e157dd94206b-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-mgxcj\" (UID: \"798f248f-d429-4f56-8752-e157dd94206b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-mgxcj"
Oct 30 21:42:15.377042 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.376998    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c69312da-8be3-4931-8a81-a8ff3f86d5e6-service-ca-bundle\") pod \"insights-operator-7c4986d449-l9b9d\" (UID: \"c69312da-8be3-4931-8a81-a8ff3f86d5e6\") " pod="openshift-insights/insights-operator-7c4986d449-l9b9d"
Oct 30 21:42:15.377042 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.377020    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c69312da-8be3-4931-8a81-a8ff3f86d5e6-serving-cert\") pod \"insights-operator-7c4986d449-l9b9d\" (UID: \"c69312da-8be3-4931-8a81-a8ff3f86d5e6\") " pod="openshift-insights/insights-operator-7c4986d449-l9b9d"
Oct 30 21:42:15.377245 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.377063    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/613e7c8a-e2c7-424a-93e0-7fa86d18a8a9-trusted-ca\") pod \"image-registry-d44c8cbb7-lmw97\" (UID: \"613e7c8a-e2c7-424a-93e0-7fa86d18a8a9\") " pod="openshift-image-registry/image-registry-d44c8cbb7-lmw97"
Oct 30 21:42:15.377245 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.377086    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/102b8049-7349-4e2b-974a-44c901eebcf3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-ns759\" (UID: \"102b8049-7349-4e2b-974a-44c901eebcf3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ns759"
Oct 30 21:42:15.377245 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.377108    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-424gm\" (UniqueName: \"kubernetes.io/projected/a32aacdc-53be-41cf-839e-435d43361890-kube-api-access-424gm\") pod \"kas-connection-checker-685895c7c9-cklcw\" (UID: \"a32aacdc-53be-41cf-839e-435d43361890\") " pod="kube-system/kas-connection-checker-685895c7c9-cklcw"
Oct 30 21:42:15.377245 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.377145    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f648e933-f035-486a-88d5-1b185c89782e-nginx-conf\") pod \"networking-console-plugin-86c478bb88-j5xsr\" (UID: \"f648e933-f035-486a-88d5-1b185c89782e\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-j5xsr"
Oct 30 21:42:15.377245 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.377172    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/613e7c8a-e2c7-424a-93e0-7fa86d18a8a9-registry-certificates\") pod \"image-registry-d44c8cbb7-lmw97\" (UID: \"613e7c8a-e2c7-424a-93e0-7fa86d18a8a9\") " pod="openshift-image-registry/image-registry-d44c8cbb7-lmw97"
Oct 30 21:42:15.377245 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.377206    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/102b8049-7349-4e2b-974a-44c901eebcf3-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-ns759\" (UID: \"102b8049-7349-4e2b-974a-44c901eebcf3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ns759"
Oct 30 21:42:15.377245 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.377228    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5e74071e-0462-42b5-a05c-d90bb638478a-config\") pod \"console-operator-7cd56b8dbc-sclrt\" (UID: \"5e74071e-0462-42b5-a05c-d90bb638478a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-sclrt"
Oct 30 21:42:15.377558 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.377253    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xk5zb\" (UniqueName: \"kubernetes.io/projected/c798fd57-e0be-4b61-860b-0b833aca7784-kube-api-access-xk5zb\") pod \"kas-connection-checker-685895c7c9-kz6dl\" (UID: \"c798fd57-e0be-4b61-860b-0b833aca7784\") " pod="kube-system/kas-connection-checker-685895c7c9-kz6dl"
Oct 30 21:42:15.377558 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.377286    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4tb7\" (UniqueName: \"kubernetes.io/projected/9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d-kube-api-access-n4tb7\") pod \"router-default-6c7f96b45-4wj99\" (UID: \"9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d\") " pod="openshift-ingress/router-default-6c7f96b45-4wj99"
Oct 30 21:42:15.377558 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.377320    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cad67bc5-c8d2-497c-ba1b-cdbae5822cfa-trusted-ca\") pod \"image-registry-b6cdbd45d-bdftd\" (UID: \"cad67bc5-c8d2-497c-ba1b-cdbae5822cfa\") " pod="openshift-image-registry/image-registry-b6cdbd45d-bdftd"
Oct 30 21:42:15.377558 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.377343    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mms6q\" (UniqueName: \"kubernetes.io/projected/84d9b682-3241-40af-87d5-f87d54ce3697-kube-api-access-mms6q\") pod \"dns-default-6pttx\" (UID: \"84d9b682-3241-40af-87d5-f87d54ce3697\") " pod="openshift-dns/dns-default-6pttx"
Oct 30 21:42:15.377558 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.377393    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cad67bc5-c8d2-497c-ba1b-cdbae5822cfa-image-registry-private-configuration\") pod \"image-registry-b6cdbd45d-bdftd\" (UID: \"cad67bc5-c8d2-497c-ba1b-cdbae5822cfa\") " pod="openshift-image-registry/image-registry-b6cdbd45d-bdftd"
Oct 30 21:42:15.377558 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.377419    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84d9b682-3241-40af-87d5-f87d54ce3697-tmp-dir\") pod \"dns-default-6pttx\" (UID: \"84d9b682-3241-40af-87d5-f87d54ce3697\") " pod="openshift-dns/dns-default-6pttx"
Oct 30 21:42:15.377558 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.377460    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqmr4\" (UniqueName: \"kubernetes.io/projected/5e74071e-0462-42b5-a05c-d90bb638478a-kube-api-access-vqmr4\") pod \"console-operator-7cd56b8dbc-sclrt\" (UID: \"5e74071e-0462-42b5-a05c-d90bb638478a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-sclrt"
Oct 30 21:42:15.377558 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.377492    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c69312da-8be3-4931-8a81-a8ff3f86d5e6-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-l9b9d\" (UID: \"c69312da-8be3-4931-8a81-a8ff3f86d5e6\") " pod="openshift-insights/insights-operator-7c4986d449-l9b9d"
Oct 30 21:42:15.377558 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.377549    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/613e7c8a-e2c7-424a-93e0-7fa86d18a8a9-bound-sa-token\") pod \"image-registry-d44c8cbb7-lmw97\" (UID: \"613e7c8a-e2c7-424a-93e0-7fa86d18a8a9\") " pod="openshift-image-registry/image-registry-d44c8cbb7-lmw97"
Oct 30 21:42:15.377964 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.377568    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/798f248f-d429-4f56-8752-e157dd94206b-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-mgxcj\" (UID: \"798f248f-d429-4f56-8752-e157dd94206b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-mgxcj"
Oct 30 21:42:15.377964 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.377573    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5e74071e-0462-42b5-a05c-d90bb638478a-serving-cert\") pod \"console-operator-7cd56b8dbc-sclrt\" (UID: \"5e74071e-0462-42b5-a05c-d90bb638478a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-sclrt"
Oct 30 21:42:15.377964 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:15.377693    2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 21:42:15.378337 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.378312    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/613e7c8a-e2c7-424a-93e0-7fa86d18a8a9-registry-certificates\") pod \"image-registry-d44c8cbb7-lmw97\" (UID: \"613e7c8a-e2c7-424a-93e0-7fa86d18a8a9\") " pod="openshift-image-registry/image-registry-d44c8cbb7-lmw97"
Oct 30 21:42:15.379785 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.379756    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/613e7c8a-e2c7-424a-93e0-7fa86d18a8a9-trusted-ca\") pod \"image-registry-d44c8cbb7-lmw97\" (UID: \"613e7c8a-e2c7-424a-93e0-7fa86d18a8a9\") " pod="openshift-image-registry/image-registry-d44c8cbb7-lmw97"
Oct 30 21:42:15.380509 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.378231    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhbrj\" (UniqueName: \"kubernetes.io/projected/d700d1eb-fbf4-45df-9b57-eca08d87ebe2-kube-api-access-nhbrj\") pod \"volume-data-source-validator-58c9f5f46d-s6427\" (UID: \"d700d1eb-fbf4-45df-9b57-eca08d87ebe2\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-s6427"
Oct 30 21:42:15.380509 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.379855    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qd5b8\" (UniqueName: \"kubernetes.io/projected/c69312da-8be3-4931-8a81-a8ff3f86d5e6-kube-api-access-qd5b8\") pod \"insights-operator-7c4986d449-l9b9d\" (UID: \"c69312da-8be3-4931-8a81-a8ff3f86d5e6\") " pod="openshift-insights/insights-operator-7c4986d449-l9b9d"
Oct 30 21:42:15.380509 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:15.379871    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/102b8049-7349-4e2b-974a-44c901eebcf3-cluster-monitoring-operator-tls podName:102b8049-7349-4e2b-974a-44c901eebcf3 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:15.879850898 +0000 UTC m=+30.690180911 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/102b8049-7349-4e2b-974a-44c901eebcf3-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-ns759" (UID: "102b8049-7349-4e2b-974a-44c901eebcf3") : secret "cluster-monitoring-operator-tls" not found
Oct 30 21:42:15.380509 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.379909    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0126a0a0-a1c8-4db3-a6af-629360e247f4-config\") pod \"service-ca-operator-544d749c8c-vqwlt\" (UID: \"0126a0a0-a1c8-4db3-a6af-629360e247f4\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vqwlt"
Oct 30 21:42:15.380509 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.379970    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d-service-ca-bundle\") pod \"router-default-6c7f96b45-4wj99\" (UID: \"9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d\") " pod="openshift-ingress/router-default-6c7f96b45-4wj99"
Oct 30 21:42:15.380509 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.380014    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/798f248f-d429-4f56-8752-e157dd94206b-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-mgxcj\" (UID: \"798f248f-d429-4f56-8752-e157dd94206b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-mgxcj"
Oct 30 21:42:15.380509 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.380040    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/613e7c8a-e2c7-424a-93e0-7fa86d18a8a9-registry-tls\") pod \"image-registry-d44c8cbb7-lmw97\" (UID: \"613e7c8a-e2c7-424a-93e0-7fa86d18a8a9\") " pod="openshift-image-registry/image-registry-d44c8cbb7-lmw97"
Oct 30 21:42:15.380509 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.380074    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/84d9b682-3241-40af-87d5-f87d54ce3697-metrics-tls\") pod \"dns-default-6pttx\" (UID: \"84d9b682-3241-40af-87d5-f87d54ce3697\") " pod="openshift-dns/dns-default-6pttx"
Oct 30 21:42:15.380509 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:15.380120    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d-service-ca-bundle podName:9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d nodeName:}" failed. No retries permitted until 2026-10-30 21:42:15.880102702 +0000 UTC m=+30.690432718 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d-service-ca-bundle") pod "router-default-6c7f96b45-4wj99" (UID: "9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d") : configmap references non-existent config key: service-ca.crt
Oct 30 21:42:15.380509 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.380125    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/613e7c8a-e2c7-424a-93e0-7fa86d18a8a9-image-registry-private-configuration\") pod \"image-registry-d44c8cbb7-lmw97\" (UID: \"613e7c8a-e2c7-424a-93e0-7fa86d18a8a9\") " pod="openshift-image-registry/image-registry-d44c8cbb7-lmw97"
Oct 30 21:42:15.380509 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:15.380196    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 21:42:15.380509 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:15.380206    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-d44c8cbb7-lmw97: secret "image-registry-tls" not found
Oct 30 21:42:15.380509 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:15.380235    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/613e7c8a-e2c7-424a-93e0-7fa86d18a8a9-registry-tls podName:613e7c8a-e2c7-424a-93e0-7fa86d18a8a9 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:15.880224934 +0000 UTC m=+30.690554935 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/613e7c8a-e2c7-424a-93e0-7fa86d18a8a9-registry-tls") pod "image-registry-d44c8cbb7-lmw97" (UID: "613e7c8a-e2c7-424a-93e0-7fa86d18a8a9") : secret "image-registry-tls" not found
Oct 30 21:42:15.380509 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.380391    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d-default-certificate\") pod \"router-default-6c7f96b45-4wj99\" (UID: \"9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d\") " pod="openshift-ingress/router-default-6c7f96b45-4wj99"
Oct 30 21:42:15.380941 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.380415    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/102b8049-7349-4e2b-974a-44c901eebcf3-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-ns759\" (UID: \"102b8049-7349-4e2b-974a-44c901eebcf3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ns759"
Oct 30 21:42:15.380941 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.380676    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/613e7c8a-e2c7-424a-93e0-7fa86d18a8a9-installation-pull-secrets\") pod \"image-registry-d44c8cbb7-lmw97\" (UID: \"613e7c8a-e2c7-424a-93e0-7fa86d18a8a9\") " pod="openshift-image-registry/image-registry-d44c8cbb7-lmw97"
Oct 30 21:42:15.380941 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.380917    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d-stats-auth\") pod \"router-default-6c7f96b45-4wj99\" (UID: \"9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d\") " pod="openshift-ingress/router-default-6c7f96b45-4wj99"
Oct 30 21:42:15.435276 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.435156    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/613e7c8a-e2c7-424a-93e0-7fa86d18a8a9-bound-sa-token\") pod \"image-registry-d44c8cbb7-lmw97\" (UID: \"613e7c8a-e2c7-424a-93e0-7fa86d18a8a9\") " pod="openshift-image-registry/image-registry-d44c8cbb7-lmw97"
Oct 30 21:42:15.435276 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.435191    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8w7g2\" (UniqueName: \"kubernetes.io/projected/798f248f-d429-4f56-8752-e157dd94206b-kube-api-access-8w7g2\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-mgxcj\" (UID: \"798f248f-d429-4f56-8752-e157dd94206b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-mgxcj"
Oct 30 21:42:15.435518 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.435464    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hw226\" (UniqueName: \"kubernetes.io/projected/613e7c8a-e2c7-424a-93e0-7fa86d18a8a9-kube-api-access-hw226\") pod \"image-registry-d44c8cbb7-lmw97\" (UID: \"613e7c8a-e2c7-424a-93e0-7fa86d18a8a9\") " pod="openshift-image-registry/image-registry-d44c8cbb7-lmw97"
Oct 30 21:42:15.435518 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.435466    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4tb7\" (UniqueName: \"kubernetes.io/projected/9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d-kube-api-access-n4tb7\") pod \"router-default-6c7f96b45-4wj99\" (UID: \"9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d\") " pod="openshift-ingress/router-default-6c7f96b45-4wj99"
Oct 30 21:42:15.435778 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.435731    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fcz7\" (UniqueName: \"kubernetes.io/projected/102b8049-7349-4e2b-974a-44c901eebcf3-kube-api-access-5fcz7\") pod \"cluster-monitoring-operator-884576c57-ns759\" (UID: \"102b8049-7349-4e2b-974a-44c901eebcf3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ns759"
Oct 30 21:42:15.435872 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.435849    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-424gm\" (UniqueName: \"kubernetes.io/projected/a32aacdc-53be-41cf-839e-435d43361890-kube-api-access-424gm\") pod \"kas-connection-checker-685895c7c9-cklcw\" (UID: \"a32aacdc-53be-41cf-839e-435d43361890\") " pod="kube-system/kas-connection-checker-685895c7c9-cklcw"
Oct 30 21:42:15.435872 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.435862    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdtc5\" (UniqueName: \"kubernetes.io/projected/3f97ee34-5dfc-454e-ba5c-afd939f3bd25-kube-api-access-cdtc5\") pod \"kas-connection-checker-685895c7c9-bgnkz\" (UID: \"3f97ee34-5dfc-454e-ba5c-afd939f3bd25\") " pod="kube-system/kas-connection-checker-685895c7c9-bgnkz"
Oct 30 21:42:15.436651 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.436630    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xk5zb\" (UniqueName: \"kubernetes.io/projected/c798fd57-e0be-4b61-860b-0b833aca7784-kube-api-access-xk5zb\") pod \"kas-connection-checker-685895c7c9-kz6dl\" (UID: \"c798fd57-e0be-4b61-860b-0b833aca7784\") " pod="kube-system/kas-connection-checker-685895c7c9-kz6dl"
Oct 30 21:42:15.437241 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.437219    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhbrj\" (UniqueName: \"kubernetes.io/projected/d700d1eb-fbf4-45df-9b57-eca08d87ebe2-kube-api-access-nhbrj\") pod \"volume-data-source-validator-58c9f5f46d-s6427\" (UID: \"d700d1eb-fbf4-45df-9b57-eca08d87ebe2\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-s6427"
Oct 30 21:42:15.437972 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.437960    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-cklcw"
Oct 30 21:42:15.481078 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.481055    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cad67bc5-c8d2-497c-ba1b-cdbae5822cfa-registry-certificates\") pod \"image-registry-b6cdbd45d-bdftd\" (UID: \"cad67bc5-c8d2-497c-ba1b-cdbae5822cfa\") " pod="openshift-image-registry/image-registry-b6cdbd45d-bdftd"
Oct 30 21:42:15.481164 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.481085    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/84d9b682-3241-40af-87d5-f87d54ce3697-config-volume\") pod \"dns-default-6pttx\" (UID: \"84d9b682-3241-40af-87d5-f87d54ce3697\") " pod="openshift-dns/dns-default-6pttx"
Oct 30 21:42:15.481164 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.481105    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0126a0a0-a1c8-4db3-a6af-629360e247f4-serving-cert\") pod \"service-ca-operator-544d749c8c-vqwlt\" (UID: \"0126a0a0-a1c8-4db3-a6af-629360e247f4\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vqwlt"
Oct 30 21:42:15.481164 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.481153    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cad67bc5-c8d2-497c-ba1b-cdbae5822cfa-registry-tls\") pod \"image-registry-b6cdbd45d-bdftd\" (UID: \"cad67bc5-c8d2-497c-ba1b-cdbae5822cfa\") " pod="openshift-image-registry/image-registry-b6cdbd45d-bdftd"
Oct 30 21:42:15.481279 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.481176    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cad67bc5-c8d2-497c-ba1b-cdbae5822cfa-ca-trust-extracted\") pod \"image-registry-b6cdbd45d-bdftd\" (UID: \"cad67bc5-c8d2-497c-ba1b-cdbae5822cfa\") " pod="openshift-image-registry/image-registry-b6cdbd45d-bdftd"
Oct 30 21:42:15.481279 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.481203    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cad67bc5-c8d2-497c-ba1b-cdbae5822cfa-installation-pull-secrets\") pod \"image-registry-b6cdbd45d-bdftd\" (UID: \"cad67bc5-c8d2-497c-ba1b-cdbae5822cfa\") " pod="openshift-image-registry/image-registry-b6cdbd45d-bdftd"
Oct 30 21:42:15.481279 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.481228    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9dpv\" (UniqueName: \"kubernetes.io/projected/cad67bc5-c8d2-497c-ba1b-cdbae5822cfa-kube-api-access-m9dpv\") pod \"image-registry-b6cdbd45d-bdftd\" (UID: \"cad67bc5-c8d2-497c-ba1b-cdbae5822cfa\") " pod="openshift-image-registry/image-registry-b6cdbd45d-bdftd"
Oct 30 21:42:15.481279 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.481251    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f648e933-f035-486a-88d5-1b185c89782e-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-j5xsr\" (UID: \"f648e933-f035-486a-88d5-1b185c89782e\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-j5xsr"
Oct 30 21:42:15.481463 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:15.481286    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 21:42:15.481463 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.481296    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c69312da-8be3-4931-8a81-a8ff3f86d5e6-service-ca-bundle\") pod \"insights-operator-7c4986d449-l9b9d\" (UID: \"c69312da-8be3-4931-8a81-a8ff3f86d5e6\") " pod="openshift-insights/insights-operator-7c4986d449-l9b9d"
Oct 30 21:42:15.481463 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:15.481303    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b6cdbd45d-bdftd: secret "image-registry-tls" not found
Oct 30 21:42:15.481463 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.481321    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c69312da-8be3-4931-8a81-a8ff3f86d5e6-serving-cert\") pod \"insights-operator-7c4986d449-l9b9d\" (UID: \"c69312da-8be3-4931-8a81-a8ff3f86d5e6\") " pod="openshift-insights/insights-operator-7c4986d449-l9b9d"
Oct 30 21:42:15.481463 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:15.481357    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cad67bc5-c8d2-497c-ba1b-cdbae5822cfa-registry-tls podName:cad67bc5-c8d2-497c-ba1b-cdbae5822cfa nodeName:}" failed. No retries permitted until 2026-10-30 21:42:15.981338955 +0000 UTC m=+30.791668972 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cad67bc5-c8d2-497c-ba1b-cdbae5822cfa-registry-tls") pod "image-registry-b6cdbd45d-bdftd" (UID: "cad67bc5-c8d2-497c-ba1b-cdbae5822cfa") : secret "image-registry-tls" not found
Oct 30 21:42:15.481463 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.481397    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f648e933-f035-486a-88d5-1b185c89782e-nginx-conf\") pod \"networking-console-plugin-86c478bb88-j5xsr\" (UID: \"f648e933-f035-486a-88d5-1b185c89782e\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-j5xsr"
Oct 30 21:42:15.481463 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.481428    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5e74071e-0462-42b5-a05c-d90bb638478a-config\") pod \"console-operator-7cd56b8dbc-sclrt\" (UID: \"5e74071e-0462-42b5-a05c-d90bb638478a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-sclrt"
Oct 30 21:42:15.481463 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.481458    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cad67bc5-c8d2-497c-ba1b-cdbae5822cfa-trusted-ca\") pod \"image-registry-b6cdbd45d-bdftd\" (UID: \"cad67bc5-c8d2-497c-ba1b-cdbae5822cfa\") " pod="openshift-image-registry/image-registry-b6cdbd45d-bdftd"
Oct 30 21:42:15.481881 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.481481    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mms6q\" (UniqueName: \"kubernetes.io/projected/84d9b682-3241-40af-87d5-f87d54ce3697-kube-api-access-mms6q\") pod \"dns-default-6pttx\" (UID: \"84d9b682-3241-40af-87d5-f87d54ce3697\") " pod="openshift-dns/dns-default-6pttx"
Oct 30 21:42:15.481881 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.481516    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cad67bc5-c8d2-497c-ba1b-cdbae5822cfa-image-registry-private-configuration\") pod \"image-registry-b6cdbd45d-bdftd\" (UID: \"cad67bc5-c8d2-497c-ba1b-cdbae5822cfa\") " pod="openshift-image-registry/image-registry-b6cdbd45d-bdftd"
Oct 30 21:42:15.481881 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.481560    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84d9b682-3241-40af-87d5-f87d54ce3697-tmp-dir\") pod \"dns-default-6pttx\" (UID: \"84d9b682-3241-40af-87d5-f87d54ce3697\") " pod="openshift-dns/dns-default-6pttx"
Oct 30 21:42:15.481881 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.481585    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqmr4\" (UniqueName: \"kubernetes.io/projected/5e74071e-0462-42b5-a05c-d90bb638478a-kube-api-access-vqmr4\") pod \"console-operator-7cd56b8dbc-sclrt\" (UID: \"5e74071e-0462-42b5-a05c-d90bb638478a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-sclrt"
Oct 30 21:42:15.481881 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.481614    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c69312da-8be3-4931-8a81-a8ff3f86d5e6-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-l9b9d\" (UID: \"c69312da-8be3-4931-8a81-a8ff3f86d5e6\") " pod="openshift-insights/insights-operator-7c4986d449-l9b9d"
Oct 30 21:42:15.481881 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.481615    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cad67bc5-c8d2-497c-ba1b-cdbae5822cfa-ca-trust-extracted\") pod \"image-registry-b6cdbd45d-bdftd\" (UID: \"cad67bc5-c8d2-497c-ba1b-cdbae5822cfa\") " pod="openshift-image-registry/image-registry-b6cdbd45d-bdftd"
Oct 30 21:42:15.481881 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.481645    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5e74071e-0462-42b5-a05c-d90bb638478a-serving-cert\") pod \"console-operator-7cd56b8dbc-sclrt\" (UID: \"5e74071e-0462-42b5-a05c-d90bb638478a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-sclrt"
Oct 30 21:42:15.481881 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.481675    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qd5b8\" (UniqueName: \"kubernetes.io/projected/c69312da-8be3-4931-8a81-a8ff3f86d5e6-kube-api-access-qd5b8\") pod \"insights-operator-7c4986d449-l9b9d\" (UID: \"c69312da-8be3-4931-8a81-a8ff3f86d5e6\") " pod="openshift-insights/insights-operator-7c4986d449-l9b9d"
Oct 30 21:42:15.481881 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.481700    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0126a0a0-a1c8-4db3-a6af-629360e247f4-config\") pod \"service-ca-operator-544d749c8c-vqwlt\" (UID: \"0126a0a0-a1c8-4db3-a6af-629360e247f4\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vqwlt"
Oct 30 21:42:15.481881 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.481724    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/84d9b682-3241-40af-87d5-f87d54ce3697-config-volume\") pod \"dns-default-6pttx\" (UID: \"84d9b682-3241-40af-87d5-f87d54ce3697\") " pod="openshift-dns/dns-default-6pttx"
Oct 30 21:42:15.481881 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.481747    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/84d9b682-3241-40af-87d5-f87d54ce3697-metrics-tls\") pod \"dns-default-6pttx\" (UID: \"84d9b682-3241-40af-87d5-f87d54ce3697\") " pod="openshift-dns/dns-default-6pttx"
Oct 30 21:42:15.481881 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.481758    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cad67bc5-c8d2-497c-ba1b-cdbae5822cfa-registry-certificates\") pod \"image-registry-b6cdbd45d-bdftd\" (UID: \"cad67bc5-c8d2-497c-ba1b-cdbae5822cfa\") " pod="openshift-image-registry/image-registry-b6cdbd45d-bdftd"
Oct 30 21:42:15.481881 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.481775    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/c69312da-8be3-4931-8a81-a8ff3f86d5e6-snapshots\") pod \"insights-operator-7c4986d449-l9b9d\" (UID: \"c69312da-8be3-4931-8a81-a8ff3f86d5e6\") " pod="openshift-insights/insights-operator-7c4986d449-l9b9d"
Oct 30 21:42:15.481881 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.481804    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l75p7\" (UniqueName: \"kubernetes.io/projected/28210ad9-18f0-4ecf-9bfc-b256440f0ce8-kube-api-access-l75p7\") pod \"ingress-canary-pzpgw\" (UID: \"28210ad9-18f0-4ecf-9bfc-b256440f0ce8\") " pod="openshift-ingress-canary/ingress-canary-pzpgw"
Oct 30 21:42:15.481881 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.481832    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cad67bc5-c8d2-497c-ba1b-cdbae5822cfa-bound-sa-token\") pod \"image-registry-b6cdbd45d-bdftd\" (UID: \"cad67bc5-c8d2-497c-ba1b-cdbae5822cfa\") " pod="openshift-image-registry/image-registry-b6cdbd45d-bdftd"
Oct 30 21:42:15.481881 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.481860    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kr7ww\" (UniqueName: \"kubernetes.io/projected/20f55ce3-5b04-4916-b529-0dd596aba68f-kube-api-access-kr7ww\") pod \"network-check-source-6f78d59f99-rmpxf\" (UID: \"20f55ce3-5b04-4916-b529-0dd596aba68f\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-rmpxf"
Oct 30 21:42:15.482756 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.481890    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/0bb51175-fa5e-4472-8d83-106bca721585-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-tlwms\" (UID: \"0bb51175-fa5e-4472-8d83-106bca721585\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-tlwms"
Oct 30 21:42:15.482756 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.481922    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c69312da-8be3-4931-8a81-a8ff3f86d5e6-tmp\") pod \"insights-operator-7c4986d449-l9b9d\" (UID: \"c69312da-8be3-4931-8a81-a8ff3f86d5e6\") " pod="openshift-insights/insights-operator-7c4986d449-l9b9d"
Oct 30 21:42:15.482756 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.481945    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5e74071e-0462-42b5-a05c-d90bb638478a-trusted-ca\") pod \"console-operator-7cd56b8dbc-sclrt\" (UID: \"5e74071e-0462-42b5-a05c-d90bb638478a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-sclrt"
Oct 30 21:42:15.482756 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.481988    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84d9b682-3241-40af-87d5-f87d54ce3697-tmp-dir\") pod \"dns-default-6pttx\" (UID: \"84d9b682-3241-40af-87d5-f87d54ce3697\") " pod="openshift-dns/dns-default-6pttx"
Oct 30 21:42:15.482756 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.481988    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0bb51175-fa5e-4472-8d83-106bca721585-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-tlwms\" (UID: \"0bb51175-fa5e-4472-8d83-106bca721585\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-tlwms"
Oct 30 21:42:15.482756 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:15.482078    2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 21:42:15.482756 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:15.482119    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f648e933-f035-486a-88d5-1b185c89782e-networking-console-plugin-cert podName:f648e933-f035-486a-88d5-1b185c89782e nodeName:}" failed. No retries permitted until 2026-10-30 21:42:15.98210601 +0000 UTC m=+30.792436028 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/f648e933-f035-486a-88d5-1b185c89782e-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-j5xsr" (UID: "f648e933-f035-486a-88d5-1b185c89782e") : secret "networking-console-plugin-cert" not found
Oct 30 21:42:15.482756 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.482522    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/c69312da-8be3-4931-8a81-a8ff3f86d5e6-snapshots\") pod \"insights-operator-7c4986d449-l9b9d\" (UID: \"c69312da-8be3-4931-8a81-a8ff3f86d5e6\") " pod="openshift-insights/insights-operator-7c4986d449-l9b9d"
Oct 30 21:42:15.482756 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.482556    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cad67bc5-c8d2-497c-ba1b-cdbae5822cfa-trusted-ca\") pod \"image-registry-b6cdbd45d-bdftd\" (UID: \"cad67bc5-c8d2-497c-ba1b-cdbae5822cfa\") " pod="openshift-image-registry/image-registry-b6cdbd45d-bdftd"
Oct 30 21:42:15.482756 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.482567    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c69312da-8be3-4931-8a81-a8ff3f86d5e6-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-l9b9d\" (UID: \"c69312da-8be3-4931-8a81-a8ff3f86d5e6\") " pod="openshift-insights/insights-operator-7c4986d449-l9b9d"
Oct 30 21:42:15.482756 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:15.482626    2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 21:42:15.482756 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:15.482685    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0bb51175-fa5e-4472-8d83-106bca721585-samples-operator-tls podName:0bb51175-fa5e-4472-8d83-106bca721585 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:15.982670734 +0000 UTC m=+30.793000749 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0bb51175-fa5e-4472-8d83-106bca721585-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-tlwms" (UID: "0bb51175-fa5e-4472-8d83-106bca721585") : secret "samples-operator-tls" not found
Oct 30 21:42:15.483342 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.482837    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c69312da-8be3-4931-8a81-a8ff3f86d5e6-tmp\") pod \"insights-operator-7c4986d449-l9b9d\" (UID: \"c69312da-8be3-4931-8a81-a8ff3f86d5e6\") " pod="openshift-insights/insights-operator-7c4986d449-l9b9d"
Oct 30 21:42:15.483342 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.483056    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/0bb51175-fa5e-4472-8d83-106bca721585-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-tlwms\" (UID: \"0bb51175-fa5e-4472-8d83-106bca721585\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-tlwms"
Oct 30 21:42:15.483342 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.483097    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdbmm\" (UniqueName: \"kubernetes.io/projected/0bb51175-fa5e-4472-8d83-106bca721585-kube-api-access-pdbmm\") pod \"cluster-samples-operator-6ddb6c8b94-tlwms\" (UID: \"0bb51175-fa5e-4472-8d83-106bca721585\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-tlwms"
Oct 30 21:42:15.483342 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.483135    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/28210ad9-18f0-4ecf-9bfc-b256440f0ce8-cert\") pod \"ingress-canary-pzpgw\" (UID: \"28210ad9-18f0-4ecf-9bfc-b256440f0ce8\") " pod="openshift-ingress-canary/ingress-canary-pzpgw"
Oct 30 21:42:15.483342 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.483168    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srwt7\" (UniqueName: \"kubernetes.io/projected/0126a0a0-a1c8-4db3-a6af-629360e247f4-kube-api-access-srwt7\") pod \"service-ca-operator-544d749c8c-vqwlt\" (UID: \"0126a0a0-a1c8-4db3-a6af-629360e247f4\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vqwlt"
Oct 30 21:42:15.483342 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.483244    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0126a0a0-a1c8-4db3-a6af-629360e247f4-config\") pod \"service-ca-operator-544d749c8c-vqwlt\" (UID: \"0126a0a0-a1c8-4db3-a6af-629360e247f4\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vqwlt"
Oct 30 21:42:15.483342 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.483294    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c69312da-8be3-4931-8a81-a8ff3f86d5e6-service-ca-bundle\") pod \"insights-operator-7c4986d449-l9b9d\" (UID: \"c69312da-8be3-4931-8a81-a8ff3f86d5e6\") " pod="openshift-insights/insights-operator-7c4986d449-l9b9d"
Oct 30 21:42:15.483671 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.483375    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5e74071e-0462-42b5-a05c-d90bb638478a-config\") pod \"console-operator-7cd56b8dbc-sclrt\" (UID: \"5e74071e-0462-42b5-a05c-d90bb638478a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-sclrt"
Oct 30 21:42:15.483671 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.483410    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5e74071e-0462-42b5-a05c-d90bb638478a-trusted-ca\") pod \"console-operator-7cd56b8dbc-sclrt\" (UID: \"5e74071e-0462-42b5-a05c-d90bb638478a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-sclrt"
Oct 30 21:42:15.483671 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:15.483515    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 21:42:15.483671 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.483555    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0126a0a0-a1c8-4db3-a6af-629360e247f4-serving-cert\") pod \"service-ca-operator-544d749c8c-vqwlt\" (UID: \"0126a0a0-a1c8-4db3-a6af-629360e247f4\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vqwlt"
Oct 30 21:42:15.483671 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:15.483587    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84d9b682-3241-40af-87d5-f87d54ce3697-metrics-tls podName:84d9b682-3241-40af-87d5-f87d54ce3697 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:15.983572567 +0000 UTC m=+30.793902580 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/84d9b682-3241-40af-87d5-f87d54ce3697-metrics-tls") pod "dns-default-6pttx" (UID: "84d9b682-3241-40af-87d5-f87d54ce3697") : secret "dns-default-metrics-tls" not found
Oct 30 21:42:15.483875 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.483847    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c69312da-8be3-4931-8a81-a8ff3f86d5e6-serving-cert\") pod \"insights-operator-7c4986d449-l9b9d\" (UID: \"c69312da-8be3-4931-8a81-a8ff3f86d5e6\") " pod="openshift-insights/insights-operator-7c4986d449-l9b9d"
Oct 30 21:42:15.483963 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.483940    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f648e933-f035-486a-88d5-1b185c89782e-nginx-conf\") pod \"networking-console-plugin-86c478bb88-j5xsr\" (UID: \"f648e933-f035-486a-88d5-1b185c89782e\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-j5xsr"
Oct 30 21:42:15.485270 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.485251    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cad67bc5-c8d2-497c-ba1b-cdbae5822cfa-image-registry-private-configuration\") pod \"image-registry-b6cdbd45d-bdftd\" (UID: \"cad67bc5-c8d2-497c-ba1b-cdbae5822cfa\") " pod="openshift-image-registry/image-registry-b6cdbd45d-bdftd"
Oct 30 21:42:15.485557 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.485520    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5e74071e-0462-42b5-a05c-d90bb638478a-serving-cert\") pod \"console-operator-7cd56b8dbc-sclrt\" (UID: \"5e74071e-0462-42b5-a05c-d90bb638478a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-sclrt"
Oct 30 21:42:15.485635 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.485617    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/28210ad9-18f0-4ecf-9bfc-b256440f0ce8-cert\") pod \"ingress-canary-pzpgw\" (UID: \"28210ad9-18f0-4ecf-9bfc-b256440f0ce8\") " pod="openshift-ingress-canary/ingress-canary-pzpgw"
Oct 30 21:42:15.485819 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.485805    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cad67bc5-c8d2-497c-ba1b-cdbae5822cfa-installation-pull-secrets\") pod \"image-registry-b6cdbd45d-bdftd\" (UID: \"cad67bc5-c8d2-497c-ba1b-cdbae5822cfa\") " pod="openshift-image-registry/image-registry-b6cdbd45d-bdftd"
Oct 30 21:42:15.488683 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.488665    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-s6427"
Oct 30 21:42:15.514607 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.514577    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-mgxcj"
Oct 30 21:42:15.539154 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.539129    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9dpv\" (UniqueName: \"kubernetes.io/projected/cad67bc5-c8d2-497c-ba1b-cdbae5822cfa-kube-api-access-m9dpv\") pod \"image-registry-b6cdbd45d-bdftd\" (UID: \"cad67bc5-c8d2-497c-ba1b-cdbae5822cfa\") " pod="openshift-image-registry/image-registry-b6cdbd45d-bdftd"
Oct 30 21:42:15.541512 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.541484    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqmr4\" (UniqueName: \"kubernetes.io/projected/5e74071e-0462-42b5-a05c-d90bb638478a-kube-api-access-vqmr4\") pod \"console-operator-7cd56b8dbc-sclrt\" (UID: \"5e74071e-0462-42b5-a05c-d90bb638478a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-sclrt"
Oct 30 21:42:15.542057 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.542034    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qd5b8\" (UniqueName: \"kubernetes.io/projected/c69312da-8be3-4931-8a81-a8ff3f86d5e6-kube-api-access-qd5b8\") pod \"insights-operator-7c4986d449-l9b9d\" (UID: \"c69312da-8be3-4931-8a81-a8ff3f86d5e6\") " pod="openshift-insights/insights-operator-7c4986d449-l9b9d"
Oct 30 21:42:15.544636 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.542775    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mms6q\" (UniqueName: \"kubernetes.io/projected/84d9b682-3241-40af-87d5-f87d54ce3697-kube-api-access-mms6q\") pod \"dns-default-6pttx\" (UID: \"84d9b682-3241-40af-87d5-f87d54ce3697\") " pod="openshift-dns/dns-default-6pttx"
Oct 30 21:42:15.544636 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.542778    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cad67bc5-c8d2-497c-ba1b-cdbae5822cfa-bound-sa-token\") pod \"image-registry-b6cdbd45d-bdftd\" (UID: \"cad67bc5-c8d2-497c-ba1b-cdbae5822cfa\") " pod="openshift-image-registry/image-registry-b6cdbd45d-bdftd"
Oct 30 21:42:15.544636 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.542879    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdbmm\" (UniqueName: \"kubernetes.io/projected/0bb51175-fa5e-4472-8d83-106bca721585-kube-api-access-pdbmm\") pod \"cluster-samples-operator-6ddb6c8b94-tlwms\" (UID: \"0bb51175-fa5e-4472-8d83-106bca721585\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-tlwms"
Oct 30 21:42:15.544636 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.542916    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kr7ww\" (UniqueName: \"kubernetes.io/projected/20f55ce3-5b04-4916-b529-0dd596aba68f-kube-api-access-kr7ww\") pod \"network-check-source-6f78d59f99-rmpxf\" (UID: \"20f55ce3-5b04-4916-b529-0dd596aba68f\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-rmpxf"
Oct 30 21:42:15.544636 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.543397    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l75p7\" (UniqueName: \"kubernetes.io/projected/28210ad9-18f0-4ecf-9bfc-b256440f0ce8-kube-api-access-l75p7\") pod \"ingress-canary-pzpgw\" (UID: \"28210ad9-18f0-4ecf-9bfc-b256440f0ce8\") " pod="openshift-ingress-canary/ingress-canary-pzpgw"
Oct 30 21:42:15.544636 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.544312    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srwt7\" (UniqueName: \"kubernetes.io/projected/0126a0a0-a1c8-4db3-a6af-629360e247f4-kube-api-access-srwt7\") pod \"service-ca-operator-544d749c8c-vqwlt\" (UID: \"0126a0a0-a1c8-4db3-a6af-629360e247f4\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vqwlt"
Oct 30 21:42:15.570769 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.567069    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vqwlt"
Oct 30 21:42:15.589825 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.589790    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-rmpxf"
Oct 30 21:42:15.612557 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.611794    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-979r9"
Oct 30 21:42:15.612557 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.612340    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6wbw4"
Oct 30 21:42:15.613069 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.612771    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q6tjc"
Oct 30 21:42:15.616664 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.616639    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-pzpgw"
Oct 30 21:42:15.663564 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.663517    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 21:42:15.665722 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.663887    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6s2r8\""
Oct 30 21:42:15.665722 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.663928    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 21:42:15.665722 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.663976    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-8c5b8\""
Oct 30 21:42:15.665722 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.664012    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-hmtv9\""
Oct 30 21:42:15.717950 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.717927    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-kz6dl"
Oct 30 21:42:15.727066 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.727045    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-bgnkz"
Oct 30 21:42:15.828652 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.828628    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-l9b9d"
Oct 30 21:42:15.840758 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.840731    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-sclrt"
Oct 30 21:42:15.882837 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.882809    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-s6427"]
Oct 30 21:42:15.884888 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.884865    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-cklcw"]
Oct 30 21:42:15.885194 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:42:15.885175    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd700d1eb_fbf4_45df_9b57_eca08d87ebe2.slice/crio-75e0ccc2236e8a6d94ca8547c5e6e12b14a26ebd6ef7344628e37e23a7e8601f WatchSource:0}: Error finding container 75e0ccc2236e8a6d94ca8547c5e6e12b14a26ebd6ef7344628e37e23a7e8601f: Status 404 returned error can't find the container with id 75e0ccc2236e8a6d94ca8547c5e6e12b14a26ebd6ef7344628e37e23a7e8601f
Oct 30 21:42:15.887926 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.887839    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d-metrics-certs\") pod \"router-default-6c7f96b45-4wj99\" (UID: \"9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d\") " pod="openshift-ingress/router-default-6c7f96b45-4wj99"
Oct 30 21:42:15.889125 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.887972    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/102b8049-7349-4e2b-974a-44c901eebcf3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-ns759\" (UID: \"102b8049-7349-4e2b-974a-44c901eebcf3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ns759"
Oct 30 21:42:15.889125 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:15.887981    2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 21:42:15.889125 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.888020    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d-service-ca-bundle\") pod \"router-default-6c7f96b45-4wj99\" (UID: \"9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d\") " pod="openshift-ingress/router-default-6c7f96b45-4wj99"
Oct 30 21:42:15.889125 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.888036    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/613e7c8a-e2c7-424a-93e0-7fa86d18a8a9-registry-tls\") pod \"image-registry-d44c8cbb7-lmw97\" (UID: \"613e7c8a-e2c7-424a-93e0-7fa86d18a8a9\") " pod="openshift-image-registry/image-registry-d44c8cbb7-lmw97"
Oct 30 21:42:15.889125 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:15.888090    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d-metrics-certs podName:9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d nodeName:}" failed. No retries permitted until 2026-10-30 21:42:16.888055018 +0000 UTC m=+31.698385035 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d-metrics-certs") pod "router-default-6c7f96b45-4wj99" (UID: "9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d") : secret "router-metrics-certs-default" not found
Oct 30 21:42:15.889125 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:15.888130    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 21:42:15.889125 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:15.888144    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-d44c8cbb7-lmw97: secret "image-registry-tls" not found
Oct 30 21:42:15.889125 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:15.888188    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/613e7c8a-e2c7-424a-93e0-7fa86d18a8a9-registry-tls podName:613e7c8a-e2c7-424a-93e0-7fa86d18a8a9 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:16.888174217 +0000 UTC m=+31.698504217 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/613e7c8a-e2c7-424a-93e0-7fa86d18a8a9-registry-tls") pod "image-registry-d44c8cbb7-lmw97" (UID: "613e7c8a-e2c7-424a-93e0-7fa86d18a8a9") : secret "image-registry-tls" not found
Oct 30 21:42:15.889125 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:15.888207    2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 21:42:15.889125 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:15.888249    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/102b8049-7349-4e2b-974a-44c901eebcf3-cluster-monitoring-operator-tls podName:102b8049-7349-4e2b-974a-44c901eebcf3 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:16.888236839 +0000 UTC m=+31.698566858 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/102b8049-7349-4e2b-974a-44c901eebcf3-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-ns759" (UID: "102b8049-7349-4e2b-974a-44c901eebcf3") : secret "cluster-monitoring-operator-tls" not found
Oct 30 21:42:15.889125 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:15.888266    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d-service-ca-bundle podName:9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d nodeName:}" failed. No retries permitted until 2026-10-30 21:42:16.888257551 +0000 UTC m=+31.698587554 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d-service-ca-bundle") pod "router-default-6c7f96b45-4wj99" (UID: "9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d") : configmap references non-existent config key: service-ca.crt
Oct 30 21:42:15.922496 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.922082    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-mgxcj"]
Oct 30 21:42:15.963493 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.962773    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-vqwlt"]
Oct 30 21:42:15.966918 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:42:15.966284    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0126a0a0_a1c8_4db3_a6af_629360e247f4.slice/crio-6fc99d3eb77d11207c5f020fdbd83137a53549d5a511203325cc56a00f45ea1c WatchSource:0}: Error finding container 6fc99d3eb77d11207c5f020fdbd83137a53549d5a511203325cc56a00f45ea1c: Status 404 returned error can't find the container with id 6fc99d3eb77d11207c5f020fdbd83137a53549d5a511203325cc56a00f45ea1c
Oct 30 21:42:15.981513 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.981126    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-rmpxf"]
Oct 30 21:42:15.989205 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.989184    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/84d9b682-3241-40af-87d5-f87d54ce3697-metrics-tls\") pod \"dns-default-6pttx\" (UID: \"84d9b682-3241-40af-87d5-f87d54ce3697\") " pod="openshift-dns/dns-default-6pttx"
Oct 30 21:42:15.989748 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.989235    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0bb51175-fa5e-4472-8d83-106bca721585-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-tlwms\" (UID: \"0bb51175-fa5e-4472-8d83-106bca721585\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-tlwms"
Oct 30 21:42:15.989748 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.989281    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cad67bc5-c8d2-497c-ba1b-cdbae5822cfa-registry-tls\") pod \"image-registry-b6cdbd45d-bdftd\" (UID: \"cad67bc5-c8d2-497c-ba1b-cdbae5822cfa\") " pod="openshift-image-registry/image-registry-b6cdbd45d-bdftd"
Oct 30 21:42:15.989748 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:15.989300    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f648e933-f035-486a-88d5-1b185c89782e-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-j5xsr\" (UID: \"f648e933-f035-486a-88d5-1b185c89782e\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-j5xsr"
Oct 30 21:42:15.989748 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:15.989427    2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 21:42:15.989748 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:15.989475    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f648e933-f035-486a-88d5-1b185c89782e-networking-console-plugin-cert podName:f648e933-f035-486a-88d5-1b185c89782e nodeName:}" failed. No retries permitted until 2026-10-30 21:42:16.989457401 +0000 UTC m=+31.799787404 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/f648e933-f035-486a-88d5-1b185c89782e-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-j5xsr" (UID: "f648e933-f035-486a-88d5-1b185c89782e") : secret "networking-console-plugin-cert" not found
Oct 30 21:42:15.989748 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:15.989541    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 21:42:15.989748 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:15.989574    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84d9b682-3241-40af-87d5-f87d54ce3697-metrics-tls podName:84d9b682-3241-40af-87d5-f87d54ce3697 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:16.989559994 +0000 UTC m=+31.799889993 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/84d9b682-3241-40af-87d5-f87d54ce3697-metrics-tls") pod "dns-default-6pttx" (UID: "84d9b682-3241-40af-87d5-f87d54ce3697") : secret "dns-default-metrics-tls" not found
Oct 30 21:42:15.989748 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:15.989610    2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 21:42:15.989748 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:15.989629    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0bb51175-fa5e-4472-8d83-106bca721585-samples-operator-tls podName:0bb51175-fa5e-4472-8d83-106bca721585 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:16.989623495 +0000 UTC m=+31.799953495 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0bb51175-fa5e-4472-8d83-106bca721585-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-tlwms" (UID: "0bb51175-fa5e-4472-8d83-106bca721585") : secret "samples-operator-tls" not found
Oct 30 21:42:15.989748 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:15.989667    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 21:42:15.989748 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:15.989682    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b6cdbd45d-bdftd: secret "image-registry-tls" not found
Oct 30 21:42:15.989748 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:15.989703    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cad67bc5-c8d2-497c-ba1b-cdbae5822cfa-registry-tls podName:cad67bc5-c8d2-497c-ba1b-cdbae5822cfa nodeName:}" failed. No retries permitted until 2026-10-30 21:42:16.989697956 +0000 UTC m=+31.800027956 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cad67bc5-c8d2-497c-ba1b-cdbae5822cfa-registry-tls") pod "image-registry-b6cdbd45d-bdftd" (UID: "cad67bc5-c8d2-497c-ba1b-cdbae5822cfa") : secret "image-registry-tls" not found
Oct 30 21:42:16.017779 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:16.017743    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-pzpgw"]
Oct 30 21:42:16.019841 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:42:16.019817    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28210ad9_18f0_4ecf_9bfc_b256440f0ce8.slice/crio-82a9465e06fce6f349977da268c5b1d09b7af9519dedc01b9ebda34e5f224703 WatchSource:0}: Error finding container 82a9465e06fce6f349977da268c5b1d09b7af9519dedc01b9ebda34e5f224703: Status 404 returned error can't find the container with id 82a9465e06fce6f349977da268c5b1d09b7af9519dedc01b9ebda34e5f224703
Oct 30 21:42:16.101577 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:16.101553    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-kz6dl"]
Oct 30 21:42:16.103776 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:42:16.103746    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc798fd57_e0be_4b61_860b_0b833aca7784.slice/crio-c5dd861e453dd124798e665be270326747c601f40e1402e9d28e79daf84c5712 WatchSource:0}: Error finding container c5dd861e453dd124798e665be270326747c601f40e1402e9d28e79daf84c5712: Status 404 returned error can't find the container with id c5dd861e453dd124798e665be270326747c601f40e1402e9d28e79daf84c5712
Oct 30 21:42:16.115556 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:16.115503    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-bgnkz"]
Oct 30 21:42:16.119337 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:42:16.119310    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f97ee34_5dfc_454e_ba5c_afd939f3bd25.slice/crio-c20b0427474a2038c6010808a9c237275e18c16b1f9fa4fa8873f8277f6abe2b WatchSource:0}: Error finding container c20b0427474a2038c6010808a9c237275e18c16b1f9fa4fa8873f8277f6abe2b: Status 404 returned error can't find the container with id c20b0427474a2038c6010808a9c237275e18c16b1f9fa4fa8873f8277f6abe2b
Oct 30 21:42:16.253076 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:16.253054    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-l9b9d"]
Oct 30 21:42:16.255659 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:42:16.255628    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc69312da_8be3_4931_8a81_a8ff3f86d5e6.slice/crio-a55e6c85fda472364c013d6264207e438910511591249633978f190007eae893 WatchSource:0}: Error finding container a55e6c85fda472364c013d6264207e438910511591249633978f190007eae893: Status 404 returned error can't find the container with id a55e6c85fda472364c013d6264207e438910511591249633978f190007eae893
Oct 30 21:42:16.291485 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:16.291424    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-sclrt"]
Oct 30 21:42:16.293637 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:42:16.293615    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e74071e_0462_42b5_a05c_d90bb638478a.slice/crio-7429a4fe42cd3da457a0e804b878c66475785612023343f677f33b799c6bf1ac WatchSource:0}: Error finding container 7429a4fe42cd3da457a0e804b878c66475785612023343f677f33b799c6bf1ac: Status 404 returned error can't find the container with id 7429a4fe42cd3da457a0e804b878c66475785612023343f677f33b799c6bf1ac
Oct 30 21:42:16.787945 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:16.787911    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-sclrt" event={"ID":"5e74071e-0462-42b5-a05c-d90bb638478a","Type":"ContainerStarted","Data":"7429a4fe42cd3da457a0e804b878c66475785612023343f677f33b799c6bf1ac"}
Oct 30 21:42:16.789325 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:16.789302    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-bgnkz" event={"ID":"3f97ee34-5dfc-454e-ba5c-afd939f3bd25","Type":"ContainerStarted","Data":"e07659f2a4f9580c4315705aae2eb86790373e874b2cc6e0f808084fed9d6181"}
Oct 30 21:42:16.789437 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:16.789347    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-bgnkz" event={"ID":"3f97ee34-5dfc-454e-ba5c-afd939f3bd25","Type":"ContainerStarted","Data":"c20b0427474a2038c6010808a9c237275e18c16b1f9fa4fa8873f8277f6abe2b"}
Oct 30 21:42:16.790506 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:16.790483    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-pzpgw" event={"ID":"28210ad9-18f0-4ecf-9bfc-b256440f0ce8","Type":"ContainerStarted","Data":"82a9465e06fce6f349977da268c5b1d09b7af9519dedc01b9ebda34e5f224703"}
Oct 30 21:42:16.791634 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:16.791613    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-rmpxf" event={"ID":"20f55ce3-5b04-4916-b529-0dd596aba68f","Type":"ContainerStarted","Data":"b1588b1f5ed94ade8da31ac44f2630d4b83ff1eb7c553d3e27b4619894d66f45"}
Oct 30 21:42:16.792936 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:16.792916    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-kz6dl" event={"ID":"c798fd57-e0be-4b61-860b-0b833aca7784","Type":"ContainerStarted","Data":"9cbfd28383cdbeb045a6b04f962a7a458393a9569473fa9d441972729ba99d7a"}
Oct 30 21:42:16.793013 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:16.792947    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-kz6dl" event={"ID":"c798fd57-e0be-4b61-860b-0b833aca7784","Type":"ContainerStarted","Data":"c5dd861e453dd124798e665be270326747c601f40e1402e9d28e79daf84c5712"}
Oct 30 21:42:16.793992 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:16.793970    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vqwlt" event={"ID":"0126a0a0-a1c8-4db3-a6af-629360e247f4","Type":"ContainerStarted","Data":"6fc99d3eb77d11207c5f020fdbd83137a53549d5a511203325cc56a00f45ea1c"}
Oct 30 21:42:16.794984 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:16.794964    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-mgxcj" event={"ID":"798f248f-d429-4f56-8752-e157dd94206b","Type":"ContainerStarted","Data":"627a6b6e27f45f5178d5ceb3ee924b28635f2e39baa1d10d1290585ff7940e73"}
Oct 30 21:42:16.795956 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:16.795927    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-s6427" event={"ID":"d700d1eb-fbf4-45df-9b57-eca08d87ebe2","Type":"ContainerStarted","Data":"75e0ccc2236e8a6d94ca8547c5e6e12b14a26ebd6ef7344628e37e23a7e8601f"}
Oct 30 21:42:16.796865 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:16.796846    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-l9b9d" event={"ID":"c69312da-8be3-4931-8a81-a8ff3f86d5e6","Type":"ContainerStarted","Data":"a55e6c85fda472364c013d6264207e438910511591249633978f190007eae893"}
Oct 30 21:42:16.798092 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:16.798071    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-cklcw" event={"ID":"a32aacdc-53be-41cf-839e-435d43361890","Type":"ContainerStarted","Data":"28576cb4fc4419fe7c134583b84067830596ac52ae2e325a9025181f8abc81a1"}
Oct 30 21:42:16.798166 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:16.798106    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-cklcw" event={"ID":"a32aacdc-53be-41cf-839e-435d43361890","Type":"ContainerStarted","Data":"d6179def9b341a17c6786ff22e0769ea943a3b99f9e3f8d3aaf621d338cb0ae6"}
Oct 30 21:42:16.899374 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:16.899352    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d-service-ca-bundle\") pod \"router-default-6c7f96b45-4wj99\" (UID: \"9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d\") " pod="openshift-ingress/router-default-6c7f96b45-4wj99"
Oct 30 21:42:16.899473 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:16.899381    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/613e7c8a-e2c7-424a-93e0-7fa86d18a8a9-registry-tls\") pod \"image-registry-d44c8cbb7-lmw97\" (UID: \"613e7c8a-e2c7-424a-93e0-7fa86d18a8a9\") " pod="openshift-image-registry/image-registry-d44c8cbb7-lmw97"
Oct 30 21:42:16.899552 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:16.899514    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d-service-ca-bundle podName:9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d nodeName:}" failed. No retries permitted until 2026-10-30 21:42:18.899495955 +0000 UTC m=+33.709825973 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d-service-ca-bundle") pod "router-default-6c7f96b45-4wj99" (UID: "9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d") : configmap references non-existent config key: service-ca.crt
Oct 30 21:42:16.899686 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:16.899585    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 21:42:16.899686 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:16.899602    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-d44c8cbb7-lmw97: secret "image-registry-tls" not found
Oct 30 21:42:16.899686 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:16.899640    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d-metrics-certs\") pod \"router-default-6c7f96b45-4wj99\" (UID: \"9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d\") " pod="openshift-ingress/router-default-6c7f96b45-4wj99"
Oct 30 21:42:16.899686 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:16.899649    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/613e7c8a-e2c7-424a-93e0-7fa86d18a8a9-registry-tls podName:613e7c8a-e2c7-424a-93e0-7fa86d18a8a9 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:18.899634075 +0000 UTC m=+33.709964075 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/613e7c8a-e2c7-424a-93e0-7fa86d18a8a9-registry-tls") pod "image-registry-d44c8cbb7-lmw97" (UID: "613e7c8a-e2c7-424a-93e0-7fa86d18a8a9") : secret "image-registry-tls" not found
Oct 30 21:42:16.899886 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:16.899720    2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 21:42:16.899886 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:16.899744    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/102b8049-7349-4e2b-974a-44c901eebcf3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-ns759\" (UID: \"102b8049-7349-4e2b-974a-44c901eebcf3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ns759"
Oct 30 21:42:16.899886 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:16.899757    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d-metrics-certs podName:9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d nodeName:}" failed. No retries permitted until 2026-10-30 21:42:18.899745693 +0000 UTC m=+33.710075696 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d-metrics-certs") pod "router-default-6c7f96b45-4wj99" (UID: "9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d") : secret "router-metrics-certs-default" not found
Oct 30 21:42:16.899886 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:16.899833    2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 21:42:16.899886 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:16.899877    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/102b8049-7349-4e2b-974a-44c901eebcf3-cluster-monitoring-operator-tls podName:102b8049-7349-4e2b-974a-44c901eebcf3 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:18.899863914 +0000 UTC m=+33.710193931 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/102b8049-7349-4e2b-974a-44c901eebcf3-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-ns759" (UID: "102b8049-7349-4e2b-974a-44c901eebcf3") : secret "cluster-monitoring-operator-tls" not found
Oct 30 21:42:17.000846 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:17.000816    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0bb51175-fa5e-4472-8d83-106bca721585-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-tlwms\" (UID: \"0bb51175-fa5e-4472-8d83-106bca721585\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-tlwms"
Oct 30 21:42:17.001332 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:17.000899    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cad67bc5-c8d2-497c-ba1b-cdbae5822cfa-registry-tls\") pod \"image-registry-b6cdbd45d-bdftd\" (UID: \"cad67bc5-c8d2-497c-ba1b-cdbae5822cfa\") " pod="openshift-image-registry/image-registry-b6cdbd45d-bdftd"
Oct 30 21:42:17.001332 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:17.000933    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f648e933-f035-486a-88d5-1b185c89782e-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-j5xsr\" (UID: \"f648e933-f035-486a-88d5-1b185c89782e\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-j5xsr"
Oct 30 21:42:17.001332 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:17.000987    2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 21:42:17.001332 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:17.001033    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 21:42:17.001332 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:17.001052    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b6cdbd45d-bdftd: secret "image-registry-tls" not found
Oct 30 21:42:17.001332 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:17.001059    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0bb51175-fa5e-4472-8d83-106bca721585-samples-operator-tls podName:0bb51175-fa5e-4472-8d83-106bca721585 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:19.001038286 +0000 UTC m=+33.811368307 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0bb51175-fa5e-4472-8d83-106bca721585-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-tlwms" (UID: "0bb51175-fa5e-4472-8d83-106bca721585") : secret "samples-operator-tls" not found
Oct 30 21:42:17.001332 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:17.001094    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cad67bc5-c8d2-497c-ba1b-cdbae5822cfa-registry-tls podName:cad67bc5-c8d2-497c-ba1b-cdbae5822cfa nodeName:}" failed. No retries permitted until 2026-10-30 21:42:19.001082306 +0000 UTC m=+33.811412306 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cad67bc5-c8d2-497c-ba1b-cdbae5822cfa-registry-tls") pod "image-registry-b6cdbd45d-bdftd" (UID: "cad67bc5-c8d2-497c-ba1b-cdbae5822cfa") : secret "image-registry-tls" not found
Oct 30 21:42:17.001332 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:17.001035    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/84d9b682-3241-40af-87d5-f87d54ce3697-metrics-tls\") pod \"dns-default-6pttx\" (UID: \"84d9b682-3241-40af-87d5-f87d54ce3697\") " pod="openshift-dns/dns-default-6pttx"
Oct 30 21:42:17.001332 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:17.001106    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 21:42:17.001332 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:17.001107    2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 21:42:17.001332 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:17.001150    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84d9b682-3241-40af-87d5-f87d54ce3697-metrics-tls podName:84d9b682-3241-40af-87d5-f87d54ce3697 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:19.00113676 +0000 UTC m=+33.811466766 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/84d9b682-3241-40af-87d5-f87d54ce3697-metrics-tls") pod "dns-default-6pttx" (UID: "84d9b682-3241-40af-87d5-f87d54ce3697") : secret "dns-default-metrics-tls" not found
Oct 30 21:42:17.001332 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:17.001166    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f648e933-f035-486a-88d5-1b185c89782e-networking-console-plugin-cert podName:f648e933-f035-486a-88d5-1b185c89782e nodeName:}" failed. No retries permitted until 2026-10-30 21:42:19.001158893 +0000 UTC m=+33.811488897 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/f648e933-f035-486a-88d5-1b185c89782e-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-j5xsr" (UID: "f648e933-f035-486a-88d5-1b185c89782e") : secret "networking-console-plugin-cert" not found
Oct 30 21:42:17.211089 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:17.210990    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-cklcw" podStartSLOduration=393.210977689 podStartE2EDuration="6m33.210977689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:35:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:17.210247539 +0000 UTC m=+32.020577562" watchObservedRunningTime="2026-10-30 21:42:17.210977689 +0000 UTC m=+32.021307710"
Oct 30 21:42:17.323362 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:17.323300    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-kz6dl" podStartSLOduration=393.323282006 podStartE2EDuration="6m33.323282006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:35:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:17.322392746 +0000 UTC m=+32.132722771" watchObservedRunningTime="2026-10-30 21:42:17.323282006 +0000 UTC m=+32.133612029"
Oct 30 21:42:18.618956 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:18.618871    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2205e8e6-f038-47fa-b393-0abe4d622e03-metrics-certs\") pod \"network-metrics-daemon-q6tjc\" (UID: \"2205e8e6-f038-47fa-b393-0abe4d622e03\") " pod="openshift-multus/network-metrics-daemon-q6tjc"
Oct 30 21:42:18.619429 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:18.618960    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fe41c70b-7e2c-432f-8389-0282d6fdb332-original-pull-secret\") pod \"global-pull-secret-syncer-6wbw4\" (UID: \"fe41c70b-7e2c-432f-8389-0282d6fdb332\") " pod="kube-system/global-pull-secret-syncer-6wbw4"
Oct 30 21:42:18.619982 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:18.619960    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 21:42:18.620075 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:18.620037    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2205e8e6-f038-47fa-b393-0abe4d622e03-metrics-certs podName:2205e8e6-f038-47fa-b393-0abe4d622e03 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:50.620015848 +0000 UTC m=+65.430345861 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2205e8e6-f038-47fa-b393-0abe4d622e03-metrics-certs") pod "network-metrics-daemon-q6tjc" (UID: "2205e8e6-f038-47fa-b393-0abe4d622e03") : secret "metrics-daemon-secret" not found
Oct 30 21:42:18.625098 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:18.625068    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fe41c70b-7e2c-432f-8389-0282d6fdb332-original-pull-secret\") pod \"global-pull-secret-syncer-6wbw4\" (UID: \"fe41c70b-7e2c-432f-8389-0282d6fdb332\") " pod="kube-system/global-pull-secret-syncer-6wbw4"
Oct 30 21:42:18.659643 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:18.659613    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6wbw4"
Oct 30 21:42:18.720187 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:18.720148    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-579v9\" (UniqueName: \"kubernetes.io/projected/3784e680-5efe-4ffb-ad71-956d05b57dce-kube-api-access-579v9\") pod \"network-check-target-979r9\" (UID: \"3784e680-5efe-4ffb-ad71-956d05b57dce\") " pod="openshift-network-diagnostics/network-check-target-979r9"
Oct 30 21:42:18.728475 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:18.728394    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-579v9\" (UniqueName: \"kubernetes.io/projected/3784e680-5efe-4ffb-ad71-956d05b57dce-kube-api-access-579v9\") pod \"network-check-target-979r9\" (UID: \"3784e680-5efe-4ffb-ad71-956d05b57dce\") " pod="openshift-network-diagnostics/network-check-target-979r9"
Oct 30 21:42:18.810959 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:18.810900    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-s6427" event={"ID":"d700d1eb-fbf4-45df-9b57-eca08d87ebe2","Type":"ContainerStarted","Data":"89003002d44ce100b0bd2e63f63e6e08e3a565adea3379679ed7c547bc0a6f22"}
Oct 30 21:42:18.921418 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:18.921245    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-bgnkz" podStartSLOduration=394.921224973 podStartE2EDuration="6m34.921224973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:35:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:17.432269412 +0000 UTC m=+32.242599434" watchObservedRunningTime="2026-10-30 21:42:18.921224973 +0000 UTC m=+33.731554996"
Oct 30 21:42:18.921883 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:18.921856    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d-metrics-certs\") pod \"router-default-6c7f96b45-4wj99\" (UID: \"9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d\") " pod="openshift-ingress/router-default-6c7f96b45-4wj99"
Oct 30 21:42:18.921999 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:18.921942    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/102b8049-7349-4e2b-974a-44c901eebcf3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-ns759\" (UID: \"102b8049-7349-4e2b-974a-44c901eebcf3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ns759"
Oct 30 21:42:18.921999 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:18.921986    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d-service-ca-bundle\") pod \"router-default-6c7f96b45-4wj99\" (UID: \"9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d\") " pod="openshift-ingress/router-default-6c7f96b45-4wj99"
Oct 30 21:42:18.922069 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:18.922002    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/613e7c8a-e2c7-424a-93e0-7fa86d18a8a9-registry-tls\") pod \"image-registry-d44c8cbb7-lmw97\" (UID: \"613e7c8a-e2c7-424a-93e0-7fa86d18a8a9\") " pod="openshift-image-registry/image-registry-d44c8cbb7-lmw97"
Oct 30 21:42:18.922121 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:18.922107    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 21:42:18.922166 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:18.922122    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-d44c8cbb7-lmw97: secret "image-registry-tls" not found
Oct 30 21:42:18.922166 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:18.922163    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/613e7c8a-e2c7-424a-93e0-7fa86d18a8a9-registry-tls podName:613e7c8a-e2c7-424a-93e0-7fa86d18a8a9 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:22.922150355 +0000 UTC m=+37.732480355 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/613e7c8a-e2c7-424a-93e0-7fa86d18a8a9-registry-tls") pod "image-registry-d44c8cbb7-lmw97" (UID: "613e7c8a-e2c7-424a-93e0-7fa86d18a8a9") : secret "image-registry-tls" not found
Oct 30 21:42:18.922230 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:18.922208    2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 21:42:18.922230 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:18.922229    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d-metrics-certs podName:9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d nodeName:}" failed. No retries permitted until 2026-10-30 21:42:22.922222752 +0000 UTC m=+37.732552752 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d-metrics-certs") pod "router-default-6c7f96b45-4wj99" (UID: "9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d") : secret "router-metrics-certs-default" not found
Oct 30 21:42:18.922330 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:18.922310    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d-service-ca-bundle podName:9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d nodeName:}" failed. No retries permitted until 2026-10-30 21:42:22.922297748 +0000 UTC m=+37.732627752 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d-service-ca-bundle") pod "router-default-6c7f96b45-4wj99" (UID: "9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d") : configmap references non-existent config key: service-ca.crt
Oct 30 21:42:18.922728 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:18.922704    2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 21:42:18.922819 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:18.922783    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/102b8049-7349-4e2b-974a-44c901eebcf3-cluster-monitoring-operator-tls podName:102b8049-7349-4e2b-974a-44c901eebcf3 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:22.922767737 +0000 UTC m=+37.733097751 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/102b8049-7349-4e2b-974a-44c901eebcf3-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-ns759" (UID: "102b8049-7349-4e2b-974a-44c901eebcf3") : secret "cluster-monitoring-operator-tls" not found
Oct 30 21:42:18.946265 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:18.946017    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-979r9"
Oct 30 21:42:19.023725 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:19.023178    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0bb51175-fa5e-4472-8d83-106bca721585-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-tlwms\" (UID: \"0bb51175-fa5e-4472-8d83-106bca721585\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-tlwms"
Oct 30 21:42:19.023725 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:19.023266    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cad67bc5-c8d2-497c-ba1b-cdbae5822cfa-registry-tls\") pod \"image-registry-b6cdbd45d-bdftd\" (UID: \"cad67bc5-c8d2-497c-ba1b-cdbae5822cfa\") " pod="openshift-image-registry/image-registry-b6cdbd45d-bdftd"
Oct 30 21:42:19.023725 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:19.023305    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f648e933-f035-486a-88d5-1b185c89782e-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-j5xsr\" (UID: \"f648e933-f035-486a-88d5-1b185c89782e\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-j5xsr"
Oct 30 21:42:19.023725 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:19.023363    2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 21:42:19.023725 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:19.023402    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/84d9b682-3241-40af-87d5-f87d54ce3697-metrics-tls\") pod \"dns-default-6pttx\" (UID: \"84d9b682-3241-40af-87d5-f87d54ce3697\") " pod="openshift-dns/dns-default-6pttx"
Oct 30 21:42:19.023725 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:19.023539    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 21:42:19.023725 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:19.023596    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84d9b682-3241-40af-87d5-f87d54ce3697-metrics-tls podName:84d9b682-3241-40af-87d5-f87d54ce3697 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:23.023579111 +0000 UTC m=+37.833909118 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/84d9b682-3241-40af-87d5-f87d54ce3697-metrics-tls") pod "dns-default-6pttx" (UID: "84d9b682-3241-40af-87d5-f87d54ce3697") : secret "dns-default-metrics-tls" not found
Oct 30 21:42:19.023725 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:19.023663    2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 21:42:19.023725 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:19.023701    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f648e933-f035-486a-88d5-1b185c89782e-networking-console-plugin-cert podName:f648e933-f035-486a-88d5-1b185c89782e nodeName:}" failed. No retries permitted until 2026-10-30 21:42:23.023689122 +0000 UTC m=+37.834019125 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/f648e933-f035-486a-88d5-1b185c89782e-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-j5xsr" (UID: "f648e933-f035-486a-88d5-1b185c89782e") : secret "networking-console-plugin-cert" not found
Oct 30 21:42:19.024271 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:19.023811    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 21:42:19.024271 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:19.023823    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b6cdbd45d-bdftd: secret "image-registry-tls" not found
Oct 30 21:42:19.024271 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:19.023862    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cad67bc5-c8d2-497c-ba1b-cdbae5822cfa-registry-tls podName:cad67bc5-c8d2-497c-ba1b-cdbae5822cfa nodeName:}" failed. No retries permitted until 2026-10-30 21:42:23.023848733 +0000 UTC m=+37.834178733 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cad67bc5-c8d2-497c-ba1b-cdbae5822cfa-registry-tls") pod "image-registry-b6cdbd45d-bdftd" (UID: "cad67bc5-c8d2-497c-ba1b-cdbae5822cfa") : secret "image-registry-tls" not found
Oct 30 21:42:19.024271 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:19.023908    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0bb51175-fa5e-4472-8d83-106bca721585-samples-operator-tls podName:0bb51175-fa5e-4472-8d83-106bca721585 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:23.02389609 +0000 UTC m=+37.834226090 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0bb51175-fa5e-4472-8d83-106bca721585-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-tlwms" (UID: "0bb51175-fa5e-4472-8d83-106bca721585") : secret "samples-operator-tls" not found
Oct 30 21:42:19.064338 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:19.064247    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-s6427" podStartSLOduration=353.749236934 podStartE2EDuration="5m56.06422537s" totalImagesPullingTime="2.314988436s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:36:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:18.921595691 +0000 UTC m=+33.731925714" watchObservedRunningTime="2026-10-30 21:42:19.06422537 +0000 UTC m=+33.874555396"
Oct 30 21:42:19.066408 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:19.066381    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6wbw4"]
Oct 30 21:42:19.357352 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:19.357322    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-979r9"]
Oct 30 21:42:19.363848 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:42:19.363819    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3784e680_5efe_4ffb_ad71_956d05b57dce.slice/crio-087dbdeaadb724f2107d6c79a0edb74fa79f9fb572925a89d6c27a8e804aa75e WatchSource:0}: Error finding container 087dbdeaadb724f2107d6c79a0edb74fa79f9fb572925a89d6c27a8e804aa75e: Status 404 returned error can't find the container with id 087dbdeaadb724f2107d6c79a0edb74fa79f9fb572925a89d6c27a8e804aa75e
Oct 30 21:42:19.816691 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:19.816612    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-979r9" event={"ID":"3784e680-5efe-4ffb-ad71-956d05b57dce","Type":"ContainerStarted","Data":"087dbdeaadb724f2107d6c79a0edb74fa79f9fb572925a89d6c27a8e804aa75e"}
Oct 30 21:42:19.820549 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:19.820441    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-6wbw4" event={"ID":"fe41c70b-7e2c-432f-8389-0282d6fdb332","Type":"ContainerStarted","Data":"65c7594af33d218097c8d7c8fa42fea0d244f99a0e390588efdeab8bdbf9ff99"}
Oct 30 21:42:20.826061 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:20.826021    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-mgxcj" event={"ID":"798f248f-d429-4f56-8752-e157dd94206b","Type":"ContainerStarted","Data":"c85648ba4fc4a16b4488b01235e3fa82e7c4592cc97d60f71a6e82d135601002"}
Oct 30 21:42:20.829837 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:20.829813    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-sclrt_5e74071e-0462-42b5-a05c-d90bb638478a/console-operator/0.log"
Oct 30 21:42:20.829958 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:20.829853    2580 generic.go:356] "Generic (PLEG): container finished" podID="5e74071e-0462-42b5-a05c-d90bb638478a" containerID="5ad34085c57820f1dc426eab6cbe15003eb8d8080f637afbd60418a61ca36317" exitCode=255
Oct 30 21:42:20.829958 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:20.829909    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-sclrt" event={"ID":"5e74071e-0462-42b5-a05c-d90bb638478a","Type":"ContainerDied","Data":"5ad34085c57820f1dc426eab6cbe15003eb8d8080f637afbd60418a61ca36317"}
Oct 30 21:42:20.830242 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:20.830210    2580 scope.go:122] "RemoveContainer" containerID="5ad34085c57820f1dc426eab6cbe15003eb8d8080f637afbd60418a61ca36317"
Oct 30 21:42:21.096168 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:21.096111    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-mgxcj" podStartSLOduration=352.671144309 podStartE2EDuration="5m57.09608997s" totalImagesPullingTime="4.424945661s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:36:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:20.98160759 +0000 UTC m=+35.791937613" watchObservedRunningTime="2026-10-30 21:42:21.09608997 +0000 UTC m=+35.906419995"
Oct 30 21:42:21.843179 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:21.843154    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vqwlt" event={"ID":"0126a0a0-a1c8-4db3-a6af-629360e247f4","Type":"ContainerStarted","Data":"cd90312306fd8b8a34ad012c7004d017560ef3e5826d1d1371868cc75a772dee"}
Oct 30 21:42:21.848046 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:21.848011    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-l9b9d" event={"ID":"c69312da-8be3-4931-8a81-a8ff3f86d5e6","Type":"ContainerStarted","Data":"ca9e120be9fab54658452c365699191ee58804aef4e46c11271dd00227394e4b"}
Oct 30 21:42:21.863569 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:21.863512    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-sclrt_5e74071e-0462-42b5-a05c-d90bb638478a/console-operator/1.log"
Oct 30 21:42:21.865638 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:21.865615    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-sclrt_5e74071e-0462-42b5-a05c-d90bb638478a/console-operator/0.log"
Oct 30 21:42:21.865739 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:21.865651    2580 generic.go:356] "Generic (PLEG): container finished" podID="5e74071e-0462-42b5-a05c-d90bb638478a" containerID="fef0614d3ea43c5da061d6b88ea8aa29855cf27a2684dc399ca63400506200bb" exitCode=255
Oct 30 21:42:21.866298 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:21.866272    2580 scope.go:122] "RemoveContainer" containerID="fef0614d3ea43c5da061d6b88ea8aa29855cf27a2684dc399ca63400506200bb"
Oct 30 21:42:21.866455 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:21.866435    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-sclrt_openshift-console-operator(5e74071e-0462-42b5-a05c-d90bb638478a)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-sclrt" podUID="5e74071e-0462-42b5-a05c-d90bb638478a"
Oct 30 21:42:21.866510 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:21.866481    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-sclrt" event={"ID":"5e74071e-0462-42b5-a05c-d90bb638478a","Type":"ContainerDied","Data":"fef0614d3ea43c5da061d6b88ea8aa29855cf27a2684dc399ca63400506200bb"}
Oct 30 21:42:21.866584 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:21.866570    2580 scope.go:122] "RemoveContainer" containerID="5ad34085c57820f1dc426eab6cbe15003eb8d8080f637afbd60418a61ca36317"
Oct 30 21:42:21.970100 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:21.969937    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vqwlt" podStartSLOduration=353.815181888 podStartE2EDuration="5m58.96991795s" totalImagesPullingTime="5.154736062s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:36:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:21.967816203 +0000 UTC m=+36.778146228" watchObservedRunningTime="2026-10-30 21:42:21.96991795 +0000 UTC m=+36.780247975"
Oct 30 21:42:22.100152 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:22.100071    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-q4khr"]
Oct 30 21:42:22.104427 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:22.103845    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-q4khr"
Oct 30 21:42:22.112514 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:22.111629    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-rznzd"]
Oct 30 21:42:22.116606 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:22.115392    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-rznzd"
Oct 30 21:42:22.154588 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:22.154547    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-xf9tm\""
Oct 30 21:42:22.154775 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:22.154692    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 30 21:42:22.154775 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:22.154729    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 30 21:42:22.163017 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:22.162989    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-q4khr"]
Oct 30 21:42:22.163499 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:22.163365    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9grk\" (UniqueName: \"kubernetes.io/projected/033356b5-3c05-4f49-ad6a-7c369259970d-kube-api-access-d9grk\") pod \"migrator-66df5d496b-q4khr\" (UID: \"033356b5-3c05-4f49-ad6a-7c369259970d\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-q4khr"
Oct 30 21:42:22.163499 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:22.163406    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4658n\" (UniqueName: \"kubernetes.io/projected/879022d4-06b9-44cc-a2a2-dc307b27345e-kube-api-access-4658n\") pod \"migrator-66df5d496b-rznzd\" (UID: \"879022d4-06b9-44cc-a2a2-dc307b27345e\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-rznzd"
Oct 30 21:42:22.173562 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:22.173434    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-rznzd"]
Oct 30 21:42:22.197520 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:22.197462    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-l9b9d" podStartSLOduration=354.602741992 podStartE2EDuration="5m59.197444302s" totalImagesPullingTime="4.59470231s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:36:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:22.19685575 +0000 UTC m=+37.007185772" watchObservedRunningTime="2026-10-30 21:42:22.197444302 +0000 UTC m=+37.007774326"
Oct 30 21:42:22.264372 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:22.264345    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9grk\" (UniqueName: \"kubernetes.io/projected/033356b5-3c05-4f49-ad6a-7c369259970d-kube-api-access-d9grk\") pod \"migrator-66df5d496b-q4khr\" (UID: \"033356b5-3c05-4f49-ad6a-7c369259970d\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-q4khr"
Oct 30 21:42:22.264562 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:22.264418    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4658n\" (UniqueName: \"kubernetes.io/projected/879022d4-06b9-44cc-a2a2-dc307b27345e-kube-api-access-4658n\") pod \"migrator-66df5d496b-rznzd\" (UID: \"879022d4-06b9-44cc-a2a2-dc307b27345e\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-rznzd"
Oct 30 21:42:22.329223 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:22.329201    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4658n\" (UniqueName: \"kubernetes.io/projected/879022d4-06b9-44cc-a2a2-dc307b27345e-kube-api-access-4658n\") pod \"migrator-66df5d496b-rznzd\" (UID: \"879022d4-06b9-44cc-a2a2-dc307b27345e\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-rznzd"
Oct 30 21:42:22.329504 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:22.329479    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9grk\" (UniqueName: \"kubernetes.io/projected/033356b5-3c05-4f49-ad6a-7c369259970d-kube-api-access-d9grk\") pod \"migrator-66df5d496b-q4khr\" (UID: \"033356b5-3c05-4f49-ad6a-7c369259970d\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-q4khr"
Oct 30 21:42:22.423170 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:22.423144    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-q4khr"
Oct 30 21:42:22.431257 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:22.431061    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-rznzd"
Oct 30 21:42:22.498787 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:22.497198    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-mstdj"]
Oct 30 21:42:22.503761 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:22.502267    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-mstdj"
Oct 30 21:42:22.552254 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:22.552170    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-mstdj"]
Oct 30 21:42:22.552840 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:22.552616    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 21:42:22.552840 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:22.552681    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 21:42:22.552840 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:22.552816    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-jxkz7\""
Oct 30 21:42:22.567293 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:22.567247    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/74ff988a-c05e-4a3e-adff-5945b79cf248-data-volume\") pod \"insights-runtime-extractor-mstdj\" (UID: \"74ff988a-c05e-4a3e-adff-5945b79cf248\") " pod="openshift-insights/insights-runtime-extractor-mstdj"
Oct 30 21:42:22.567425 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:22.567390    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67qtb\" (UniqueName: \"kubernetes.io/projected/74ff988a-c05e-4a3e-adff-5945b79cf248-kube-api-access-67qtb\") pod \"insights-runtime-extractor-mstdj\" (UID: \"74ff988a-c05e-4a3e-adff-5945b79cf248\") " pod="openshift-insights/insights-runtime-extractor-mstdj"
Oct 30 21:42:22.567604 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:22.567582    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/74ff988a-c05e-4a3e-adff-5945b79cf248-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mstdj\" (UID: \"74ff988a-c05e-4a3e-adff-5945b79cf248\") " pod="openshift-insights/insights-runtime-extractor-mstdj"
Oct 30 21:42:22.567657 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:22.567624    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/74ff988a-c05e-4a3e-adff-5945b79cf248-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mstdj\" (UID: \"74ff988a-c05e-4a3e-adff-5945b79cf248\") " pod="openshift-insights/insights-runtime-extractor-mstdj"
Oct 30 21:42:22.567825 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:22.567804    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/74ff988a-c05e-4a3e-adff-5945b79cf248-crio-socket\") pod \"insights-runtime-extractor-mstdj\" (UID: \"74ff988a-c05e-4a3e-adff-5945b79cf248\") " pod="openshift-insights/insights-runtime-extractor-mstdj"
Oct 30 21:42:22.668507 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:22.668411    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/74ff988a-c05e-4a3e-adff-5945b79cf248-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mstdj\" (UID: \"74ff988a-c05e-4a3e-adff-5945b79cf248\") " pod="openshift-insights/insights-runtime-extractor-mstdj"
Oct 30 21:42:22.668507 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:22.668458    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/74ff988a-c05e-4a3e-adff-5945b79cf248-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mstdj\" (UID: \"74ff988a-c05e-4a3e-adff-5945b79cf248\") " pod="openshift-insights/insights-runtime-extractor-mstdj"
Oct 30 21:42:22.668754 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:22.668564    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 21:42:22.668754 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:22.668631    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/74ff988a-c05e-4a3e-adff-5945b79cf248-kube-rbac-proxy-cm podName:74ff988a-c05e-4a3e-adff-5945b79cf248 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:23.168608537 +0000 UTC m=+37.978938552 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/74ff988a-c05e-4a3e-adff-5945b79cf248-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mstdj" (UID: "74ff988a-c05e-4a3e-adff-5945b79cf248") : configmap "kube-rbac-proxy" not found
Oct 30 21:42:22.668754 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:22.668649    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/74ff988a-c05e-4a3e-adff-5945b79cf248-crio-socket\") pod \"insights-runtime-extractor-mstdj\" (UID: \"74ff988a-c05e-4a3e-adff-5945b79cf248\") " pod="openshift-insights/insights-runtime-extractor-mstdj"
Oct 30 21:42:22.668754 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:22.668660    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 21:42:22.668754 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:22.668679    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/74ff988a-c05e-4a3e-adff-5945b79cf248-data-volume\") pod \"insights-runtime-extractor-mstdj\" (UID: \"74ff988a-c05e-4a3e-adff-5945b79cf248\") " pod="openshift-insights/insights-runtime-extractor-mstdj"
Oct 30 21:42:22.668754 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:22.668723    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74ff988a-c05e-4a3e-adff-5945b79cf248-insights-runtime-extractor-tls podName:74ff988a-c05e-4a3e-adff-5945b79cf248 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:23.16870608 +0000 UTC m=+37.979036084 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/74ff988a-c05e-4a3e-adff-5945b79cf248-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mstdj" (UID: "74ff988a-c05e-4a3e-adff-5945b79cf248") : secret "insights-runtime-extractor-tls" not found
Oct 30 21:42:22.669022 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:22.668780    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67qtb\" (UniqueName: \"kubernetes.io/projected/74ff988a-c05e-4a3e-adff-5945b79cf248-kube-api-access-67qtb\") pod \"insights-runtime-extractor-mstdj\" (UID: \"74ff988a-c05e-4a3e-adff-5945b79cf248\") " pod="openshift-insights/insights-runtime-extractor-mstdj"
Oct 30 21:42:22.669022 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:22.668989    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/74ff988a-c05e-4a3e-adff-5945b79cf248-crio-socket\") pod \"insights-runtime-extractor-mstdj\" (UID: \"74ff988a-c05e-4a3e-adff-5945b79cf248\") " pod="openshift-insights/insights-runtime-extractor-mstdj"
Oct 30 21:42:22.669022 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:22.669001    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/74ff988a-c05e-4a3e-adff-5945b79cf248-data-volume\") pod \"insights-runtime-extractor-mstdj\" (UID: \"74ff988a-c05e-4a3e-adff-5945b79cf248\") " pod="openshift-insights/insights-runtime-extractor-mstdj"
Oct 30 21:42:22.694967 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:22.694931    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-7t4kc-b65p7"]
Oct 30 21:42:22.698718 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:22.698687    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-7t4kc-b65p7"
Oct 30 21:42:22.728141 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:22.728109    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67qtb\" (UniqueName: \"kubernetes.io/projected/74ff988a-c05e-4a3e-adff-5945b79cf248-kube-api-access-67qtb\") pod \"insights-runtime-extractor-mstdj\" (UID: \"74ff988a-c05e-4a3e-adff-5945b79cf248\") " pod="openshift-insights/insights-runtime-extractor-mstdj"
Oct 30 21:42:22.755329 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:22.755263    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-7t4kc-b65p7"]
Oct 30 21:42:22.769613 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:22.769192    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xm5l5\" (UniqueName: \"kubernetes.io/projected/101c697b-d7b8-4cfe-9d4c-818b5c04108f-kube-api-access-xm5l5\") pod \"periodic-gathering-7t4kc-b65p7\" (UID: \"101c697b-d7b8-4cfe-9d4c-818b5c04108f\") " pod="openshift-insights/periodic-gathering-7t4kc-b65p7"
Oct 30 21:42:22.769613 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:22.769319    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/101c697b-d7b8-4cfe-9d4c-818b5c04108f-service-ca-bundle\") pod \"periodic-gathering-7t4kc-b65p7\" (UID: \"101c697b-d7b8-4cfe-9d4c-818b5c04108f\") " pod="openshift-insights/periodic-gathering-7t4kc-b65p7"
Oct 30 21:42:22.769613 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:22.769360    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/101c697b-d7b8-4cfe-9d4c-818b5c04108f-archives-path\") pod \"periodic-gathering-7t4kc-b65p7\" (UID: \"101c697b-d7b8-4cfe-9d4c-818b5c04108f\") " pod="openshift-insights/periodic-gathering-7t4kc-b65p7"
Oct 30 21:42:22.769613 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:22.769389    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/101c697b-d7b8-4cfe-9d4c-818b5c04108f-trusted-ca-bundle\") pod \"periodic-gathering-7t4kc-b65p7\" (UID: \"101c697b-d7b8-4cfe-9d4c-818b5c04108f\") " pod="openshift-insights/periodic-gathering-7t4kc-b65p7"
Oct 30 21:42:22.804732 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:22.804702    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-q4khr"]
Oct 30 21:42:22.810441 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:42:22.810419    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod033356b5_3c05_4f49_ad6a_7c369259970d.slice/crio-4b41219f99fde72e7b604b8387125ad69cba783b893bfd96f2e27d0ff43c8716 WatchSource:0}: Error finding container 4b41219f99fde72e7b604b8387125ad69cba783b893bfd96f2e27d0ff43c8716: Status 404 returned error can't find the container with id 4b41219f99fde72e7b604b8387125ad69cba783b893bfd96f2e27d0ff43c8716
Oct 30 21:42:22.854626 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:22.854561    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-rznzd"]
Oct 30 21:42:22.859130 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:42:22.859089    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod879022d4_06b9_44cc_a2a2_dc307b27345e.slice/crio-fd3c3fb68c3b02cc83aea080da102d52022e8b0f93fdcaa3578ed0d47045be84 WatchSource:0}: Error finding container fd3c3fb68c3b02cc83aea080da102d52022e8b0f93fdcaa3578ed0d47045be84: Status 404 returned error can't find the container with id fd3c3fb68c3b02cc83aea080da102d52022e8b0f93fdcaa3578ed0d47045be84
Oct 30 21:42:22.869858 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:22.869834    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/101c697b-d7b8-4cfe-9d4c-818b5c04108f-service-ca-bundle\") pod \"periodic-gathering-7t4kc-b65p7\" (UID: \"101c697b-d7b8-4cfe-9d4c-818b5c04108f\") " pod="openshift-insights/periodic-gathering-7t4kc-b65p7"
Oct 30 21:42:22.869963 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:22.869881    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/101c697b-d7b8-4cfe-9d4c-818b5c04108f-archives-path\") pod \"periodic-gathering-7t4kc-b65p7\" (UID: \"101c697b-d7b8-4cfe-9d4c-818b5c04108f\") " pod="openshift-insights/periodic-gathering-7t4kc-b65p7"
Oct 30 21:42:22.869963 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:22.869924    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/101c697b-d7b8-4cfe-9d4c-818b5c04108f-trusted-ca-bundle\") pod \"periodic-gathering-7t4kc-b65p7\" (UID: \"101c697b-d7b8-4cfe-9d4c-818b5c04108f\") " pod="openshift-insights/periodic-gathering-7t4kc-b65p7"
Oct 30 21:42:22.873143 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:22.870059    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xm5l5\" (UniqueName: \"kubernetes.io/projected/101c697b-d7b8-4cfe-9d4c-818b5c04108f-kube-api-access-xm5l5\") pod \"periodic-gathering-7t4kc-b65p7\" (UID: \"101c697b-d7b8-4cfe-9d4c-818b5c04108f\") " pod="openshift-insights/periodic-gathering-7t4kc-b65p7"
Oct 30 21:42:22.873143 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:22.871186    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/101c697b-d7b8-4cfe-9d4c-818b5c04108f-service-ca-bundle\") pod \"periodic-gathering-7t4kc-b65p7\" (UID: \"101c697b-d7b8-4cfe-9d4c-818b5c04108f\") " pod="openshift-insights/periodic-gathering-7t4kc-b65p7"
Oct 30 21:42:22.873143 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:22.871459    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/101c697b-d7b8-4cfe-9d4c-818b5c04108f-archives-path\") pod \"periodic-gathering-7t4kc-b65p7\" (UID: \"101c697b-d7b8-4cfe-9d4c-818b5c04108f\") " pod="openshift-insights/periodic-gathering-7t4kc-b65p7"
Oct 30 21:42:22.873143 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:22.872714    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/101c697b-d7b8-4cfe-9d4c-818b5c04108f-trusted-ca-bundle\") pod \"periodic-gathering-7t4kc-b65p7\" (UID: \"101c697b-d7b8-4cfe-9d4c-818b5c04108f\") " pod="openshift-insights/periodic-gathering-7t4kc-b65p7"
Oct 30 21:42:22.875664 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:22.875644    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-sclrt_5e74071e-0462-42b5-a05c-d90bb638478a/console-operator/1.log"
Oct 30 21:42:22.876099 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:22.875969    2580 scope.go:122] "RemoveContainer" containerID="fef0614d3ea43c5da061d6b88ea8aa29855cf27a2684dc399ca63400506200bb"
Oct 30 21:42:22.876209 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:22.876178    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-sclrt_openshift-console-operator(5e74071e-0462-42b5-a05c-d90bb638478a)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-sclrt" podUID="5e74071e-0462-42b5-a05c-d90bb638478a"
Oct 30 21:42:22.878686 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:22.878662    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-pzpgw" event={"ID":"28210ad9-18f0-4ecf-9bfc-b256440f0ce8","Type":"ContainerStarted","Data":"92335d47ddee435de0322aa6a8f9a68bad014f538dbc0f83768deaa38865b1fe"}
Oct 30 21:42:22.880924 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:22.880902    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-rmpxf" event={"ID":"20f55ce3-5b04-4916-b529-0dd596aba68f","Type":"ContainerStarted","Data":"f06e9b1f009ab5409ac3cb92a4563a5ac30ce3b168fd3515d12fe88fe9511ccd"}
Oct 30 21:42:22.882540 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:22.882504    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-q4khr" event={"ID":"033356b5-3c05-4f49-ad6a-7c369259970d","Type":"ContainerStarted","Data":"4b41219f99fde72e7b604b8387125ad69cba783b893bfd96f2e27d0ff43c8716"}
Oct 30 21:42:22.883996 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:22.883971    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-rznzd" event={"ID":"879022d4-06b9-44cc-a2a2-dc307b27345e","Type":"ContainerStarted","Data":"fd3c3fb68c3b02cc83aea080da102d52022e8b0f93fdcaa3578ed0d47045be84"}
Oct 30 21:42:22.938261 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:22.938179    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xm5l5\" (UniqueName: \"kubernetes.io/projected/101c697b-d7b8-4cfe-9d4c-818b5c04108f-kube-api-access-xm5l5\") pod \"periodic-gathering-7t4kc-b65p7\" (UID: \"101c697b-d7b8-4cfe-9d4c-818b5c04108f\") " pod="openshift-insights/periodic-gathering-7t4kc-b65p7"
Oct 30 21:42:22.973560 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:22.971890    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/102b8049-7349-4e2b-974a-44c901eebcf3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-ns759\" (UID: \"102b8049-7349-4e2b-974a-44c901eebcf3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ns759"
Oct 30 21:42:22.973560 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:22.972018    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d-service-ca-bundle\") pod \"router-default-6c7f96b45-4wj99\" (UID: \"9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d\") " pod="openshift-ingress/router-default-6c7f96b45-4wj99"
Oct 30 21:42:22.973560 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:22.972037    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/613e7c8a-e2c7-424a-93e0-7fa86d18a8a9-registry-tls\") pod \"image-registry-d44c8cbb7-lmw97\" (UID: \"613e7c8a-e2c7-424a-93e0-7fa86d18a8a9\") " pod="openshift-image-registry/image-registry-d44c8cbb7-lmw97"
Oct 30 21:42:22.973560 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:22.972121    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d-metrics-certs\") pod \"router-default-6c7f96b45-4wj99\" (UID: \"9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d\") " pod="openshift-ingress/router-default-6c7f96b45-4wj99"
Oct 30 21:42:22.973560 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:22.972224    2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 21:42:22.973560 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:22.972270    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d-metrics-certs podName:9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d nodeName:}" failed. No retries permitted until 2026-10-30 21:42:30.972255477 +0000 UTC m=+45.782585490 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d-metrics-certs") pod "router-default-6c7f96b45-4wj99" (UID: "9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d") : secret "router-metrics-certs-default" not found
Oct 30 21:42:22.973560 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:22.973288    2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 21:42:22.973560 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:22.973334    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/102b8049-7349-4e2b-974a-44c901eebcf3-cluster-monitoring-operator-tls podName:102b8049-7349-4e2b-974a-44c901eebcf3 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:30.973318868 +0000 UTC m=+45.783648884 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/102b8049-7349-4e2b-974a-44c901eebcf3-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-ns759" (UID: "102b8049-7349-4e2b-974a-44c901eebcf3") : secret "cluster-monitoring-operator-tls" not found
Oct 30 21:42:22.975409 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:22.974118    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d-service-ca-bundle podName:9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d nodeName:}" failed. No retries permitted until 2026-10-30 21:42:30.974101171 +0000 UTC m=+45.784431186 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d-service-ca-bundle") pod "router-default-6c7f96b45-4wj99" (UID: "9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d") : configmap references non-existent config key: service-ca.crt
Oct 30 21:42:22.975409 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:22.975047    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 21:42:22.975409 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:22.975068    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-d44c8cbb7-lmw97: secret "image-registry-tls" not found
Oct 30 21:42:22.975409 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:22.975117    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/613e7c8a-e2c7-424a-93e0-7fa86d18a8a9-registry-tls podName:613e7c8a-e2c7-424a-93e0-7fa86d18a8a9 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:30.975100404 +0000 UTC m=+45.785430426 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/613e7c8a-e2c7-424a-93e0-7fa86d18a8a9-registry-tls") pod "image-registry-d44c8cbb7-lmw97" (UID: "613e7c8a-e2c7-424a-93e0-7fa86d18a8a9") : secret "image-registry-tls" not found
Oct 30 21:42:23.012370 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:23.012341    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-7t4kc-b65p7"
Oct 30 21:42:23.074433 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:23.073580    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cad67bc5-c8d2-497c-ba1b-cdbae5822cfa-registry-tls\") pod \"image-registry-b6cdbd45d-bdftd\" (UID: \"cad67bc5-c8d2-497c-ba1b-cdbae5822cfa\") " pod="openshift-image-registry/image-registry-b6cdbd45d-bdftd"
Oct 30 21:42:23.074433 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:23.073635    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f648e933-f035-486a-88d5-1b185c89782e-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-j5xsr\" (UID: \"f648e933-f035-486a-88d5-1b185c89782e\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-j5xsr"
Oct 30 21:42:23.074433 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:23.073743    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/84d9b682-3241-40af-87d5-f87d54ce3697-metrics-tls\") pod \"dns-default-6pttx\" (UID: \"84d9b682-3241-40af-87d5-f87d54ce3697\") " pod="openshift-dns/dns-default-6pttx"
Oct 30 21:42:23.074433 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:23.073833    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0bb51175-fa5e-4472-8d83-106bca721585-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-tlwms\" (UID: \"0bb51175-fa5e-4472-8d83-106bca721585\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-tlwms"
Oct 30 21:42:23.074433 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:23.073995    2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 21:42:23.074433 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:23.074308    2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 21:42:23.074433 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:23.074420    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 21:42:23.074433 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:23.074433    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b6cdbd45d-bdftd: secret "image-registry-tls" not found
Oct 30 21:42:23.075071 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:23.074508    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 21:42:23.075873 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:23.075791    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0bb51175-fa5e-4472-8d83-106bca721585-samples-operator-tls podName:0bb51175-fa5e-4472-8d83-106bca721585 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:31.074042314 +0000 UTC m=+45.884372331 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0bb51175-fa5e-4472-8d83-106bca721585-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-tlwms" (UID: "0bb51175-fa5e-4472-8d83-106bca721585") : secret "samples-operator-tls" not found
Oct 30 21:42:23.075873 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:23.075823    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f648e933-f035-486a-88d5-1b185c89782e-networking-console-plugin-cert podName:f648e933-f035-486a-88d5-1b185c89782e nodeName:}" failed. No retries permitted until 2026-10-30 21:42:31.075808947 +0000 UTC m=+45.886138961 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/f648e933-f035-486a-88d5-1b185c89782e-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-j5xsr" (UID: "f648e933-f035-486a-88d5-1b185c89782e") : secret "networking-console-plugin-cert" not found
Oct 30 21:42:23.075873 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:23.075839    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cad67bc5-c8d2-497c-ba1b-cdbae5822cfa-registry-tls podName:cad67bc5-c8d2-497c-ba1b-cdbae5822cfa nodeName:}" failed. No retries permitted until 2026-10-30 21:42:31.075830475 +0000 UTC m=+45.886160476 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cad67bc5-c8d2-497c-ba1b-cdbae5822cfa-registry-tls") pod "image-registry-b6cdbd45d-bdftd" (UID: "cad67bc5-c8d2-497c-ba1b-cdbae5822cfa") : secret "image-registry-tls" not found
Oct 30 21:42:23.075873 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:23.075854    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84d9b682-3241-40af-87d5-f87d54ce3697-metrics-tls podName:84d9b682-3241-40af-87d5-f87d54ce3697 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:31.075845774 +0000 UTC m=+45.886175843 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/84d9b682-3241-40af-87d5-f87d54ce3697-metrics-tls") pod "dns-default-6pttx" (UID: "84d9b682-3241-40af-87d5-f87d54ce3697") : secret "dns-default-metrics-tls" not found
Oct 30 21:42:23.104003 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:23.103949    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-pzpgw" podStartSLOduration=2.310463171 podStartE2EDuration="8.103929887s" totalImagesPullingTime="5.793466716s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:42:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:23.103494022 +0000 UTC m=+37.913824044" watchObservedRunningTime="2026-10-30 21:42:23.103929887 +0000 UTC m=+37.914259910"
Oct 30 21:42:23.175395 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:23.175362    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/74ff988a-c05e-4a3e-adff-5945b79cf248-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mstdj\" (UID: \"74ff988a-c05e-4a3e-adff-5945b79cf248\") " pod="openshift-insights/insights-runtime-extractor-mstdj"
Oct 30 21:42:23.175559 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:23.175403    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/74ff988a-c05e-4a3e-adff-5945b79cf248-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mstdj\" (UID: \"74ff988a-c05e-4a3e-adff-5945b79cf248\") " pod="openshift-insights/insights-runtime-extractor-mstdj"
Oct 30 21:42:23.175559 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:23.175514    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 21:42:23.175674 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:23.175589    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/74ff988a-c05e-4a3e-adff-5945b79cf248-kube-rbac-proxy-cm podName:74ff988a-c05e-4a3e-adff-5945b79cf248 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:24.175576196 +0000 UTC m=+38.985906200 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/74ff988a-c05e-4a3e-adff-5945b79cf248-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mstdj" (UID: "74ff988a-c05e-4a3e-adff-5945b79cf248") : configmap "kube-rbac-proxy" not found
Oct 30 21:42:23.175674 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:23.175652    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 21:42:23.175784 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:23.175680    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74ff988a-c05e-4a3e-adff-5945b79cf248-insights-runtime-extractor-tls podName:74ff988a-c05e-4a3e-adff-5945b79cf248 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:24.17567295 +0000 UTC m=+38.986002951 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/74ff988a-c05e-4a3e-adff-5945b79cf248-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mstdj" (UID: "74ff988a-c05e-4a3e-adff-5945b79cf248") : secret "insights-runtime-extractor-tls" not found
Oct 30 21:42:23.212203 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:23.212155    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-rmpxf" podStartSLOduration=352.895159945 podStartE2EDuration="5m59.212141233s" totalImagesPullingTime="6.316981288s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:36:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:23.211658957 +0000 UTC m=+38.021988980" watchObservedRunningTime="2026-10-30 21:42:23.212141233 +0000 UTC m=+38.022471255"
Oct 30 21:42:23.389973 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:23.389938    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-7t4kc-b65p7"]
Oct 30 21:42:23.393257 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:42:23.393230    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod101c697b_d7b8_4cfe_9d4c_818b5c04108f.slice/crio-c25682648a4da3674e83bed18c8a7c4f12a24442876464de11bece9d61180c06 WatchSource:0}: Error finding container c25682648a4da3674e83bed18c8a7c4f12a24442876464de11bece9d61180c06: Status 404 returned error can't find the container with id c25682648a4da3674e83bed18c8a7c4f12a24442876464de11bece9d61180c06
Oct 30 21:42:23.891166 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:23.890933    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-979r9" event={"ID":"3784e680-5efe-4ffb-ad71-956d05b57dce","Type":"ContainerStarted","Data":"378688d232055e2b75f24c9d495d9b7c6e35ebc43b67ca818c7f9eee10195a1f"}
Oct 30 21:42:23.891166 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:23.891155    2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-979r9"
Oct 30 21:42:23.893175 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:23.893145    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-7t4kc-b65p7" event={"ID":"101c697b-d7b8-4cfe-9d4c-818b5c04108f","Type":"ContainerStarted","Data":"b8fc53580d7eec0aa43f21a2e71f286fdde0bb9b422c8c9afdbfee8ea539aadc"}
Oct 30 21:42:23.893280 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:23.893196    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-7t4kc-b65p7" event={"ID":"101c697b-d7b8-4cfe-9d4c-818b5c04108f","Type":"ContainerStarted","Data":"c25682648a4da3674e83bed18c8a7c4f12a24442876464de11bece9d61180c06"}
Oct 30 21:42:23.943503 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:23.943469    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-th574"]
Oct 30 21:42:23.948060 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:23.948036    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-th574"
Oct 30 21:42:23.985385 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:23.985352    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zv2qw\" (UniqueName: \"kubernetes.io/projected/c2386477-903e-4c6c-b807-3f473a58a6b6-kube-api-access-zv2qw\") pod \"service-ca-5fffc97bdf-th574\" (UID: \"c2386477-903e-4c6c-b807-3f473a58a6b6\") " pod="openshift-service-ca/service-ca-5fffc97bdf-th574"
Oct 30 21:42:23.985510 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:23.985420    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/c2386477-903e-4c6c-b807-3f473a58a6b6-signing-key\") pod \"service-ca-5fffc97bdf-th574\" (UID: \"c2386477-903e-4c6c-b807-3f473a58a6b6\") " pod="openshift-service-ca/service-ca-5fffc97bdf-th574"
Oct 30 21:42:23.985645 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:23.985574    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/c2386477-903e-4c6c-b807-3f473a58a6b6-signing-cabundle\") pod \"service-ca-5fffc97bdf-th574\" (UID: \"c2386477-903e-4c6c-b807-3f473a58a6b6\") " pod="openshift-service-ca/service-ca-5fffc97bdf-th574"
Oct 30 21:42:23.998589 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:23.998568    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-twk8z\""
Oct 30 21:42:23.998589 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:23.998580    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Oct 30 21:42:24.002808 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:24.002761    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-979r9" podStartSLOduration=34.177220097 podStartE2EDuration="38.002746365s" totalImagesPullingTime="3.825526268s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:41:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:24.002102271 +0000 UTC m=+38.812432294" watchObservedRunningTime="2026-10-30 21:42:24.002746365 +0000 UTC m=+38.813076389"
Oct 30 21:42:24.002904 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:24.002818    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Oct 30 21:42:24.004873 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:24.004855    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Oct 30 21:42:24.004954 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:24.004933    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Oct 30 21:42:24.008248 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:24.008225    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-th574"]
Oct 30 21:42:24.086783 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:24.086714    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/c2386477-903e-4c6c-b807-3f473a58a6b6-signing-cabundle\") pod \"service-ca-5fffc97bdf-th574\" (UID: \"c2386477-903e-4c6c-b807-3f473a58a6b6\") " pod="openshift-service-ca/service-ca-5fffc97bdf-th574"
Oct 30 21:42:24.086958 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:24.086842    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zv2qw\" (UniqueName: \"kubernetes.io/projected/c2386477-903e-4c6c-b807-3f473a58a6b6-kube-api-access-zv2qw\") pod \"service-ca-5fffc97bdf-th574\" (UID: \"c2386477-903e-4c6c-b807-3f473a58a6b6\") " pod="openshift-service-ca/service-ca-5fffc97bdf-th574"
Oct 30 21:42:24.086958 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:24.086885    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/c2386477-903e-4c6c-b807-3f473a58a6b6-signing-key\") pod \"service-ca-5fffc97bdf-th574\" (UID: \"c2386477-903e-4c6c-b807-3f473a58a6b6\") " pod="openshift-service-ca/service-ca-5fffc97bdf-th574"
Oct 30 21:42:24.087549 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:24.087500    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/c2386477-903e-4c6c-b807-3f473a58a6b6-signing-cabundle\") pod \"service-ca-5fffc97bdf-th574\" (UID: \"c2386477-903e-4c6c-b807-3f473a58a6b6\") " pod="openshift-service-ca/service-ca-5fffc97bdf-th574"
Oct 30 21:42:24.091002 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:24.090976    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/c2386477-903e-4c6c-b807-3f473a58a6b6-signing-key\") pod \"service-ca-5fffc97bdf-th574\" (UID: \"c2386477-903e-4c6c-b807-3f473a58a6b6\") " pod="openshift-service-ca/service-ca-5fffc97bdf-th574"
Oct 30 21:42:24.145095 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:24.145058    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zv2qw\" (UniqueName: \"kubernetes.io/projected/c2386477-903e-4c6c-b807-3f473a58a6b6-kube-api-access-zv2qw\") pod \"service-ca-5fffc97bdf-th574\" (UID: \"c2386477-903e-4c6c-b807-3f473a58a6b6\") " pod="openshift-service-ca/service-ca-5fffc97bdf-th574"
Oct 30 21:42:24.188471 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:24.188432    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/74ff988a-c05e-4a3e-adff-5945b79cf248-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mstdj\" (UID: \"74ff988a-c05e-4a3e-adff-5945b79cf248\") " pod="openshift-insights/insights-runtime-extractor-mstdj"
Oct 30 21:42:24.188630 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:24.188486    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/74ff988a-c05e-4a3e-adff-5945b79cf248-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mstdj\" (UID: \"74ff988a-c05e-4a3e-adff-5945b79cf248\") " pod="openshift-insights/insights-runtime-extractor-mstdj"
Oct 30 21:42:24.188710 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:24.188694    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 21:42:24.188772 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:24.188758    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74ff988a-c05e-4a3e-adff-5945b79cf248-insights-runtime-extractor-tls podName:74ff988a-c05e-4a3e-adff-5945b79cf248 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:26.188740367 +0000 UTC m=+40.999070373 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/74ff988a-c05e-4a3e-adff-5945b79cf248-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mstdj" (UID: "74ff988a-c05e-4a3e-adff-5945b79cf248") : secret "insights-runtime-extractor-tls" not found
Oct 30 21:42:24.189127 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:24.189100    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 21:42:24.189246 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:24.189157    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/74ff988a-c05e-4a3e-adff-5945b79cf248-kube-rbac-proxy-cm podName:74ff988a-c05e-4a3e-adff-5945b79cf248 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:26.189143265 +0000 UTC m=+40.999473268 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/74ff988a-c05e-4a3e-adff-5945b79cf248-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mstdj" (UID: "74ff988a-c05e-4a3e-adff-5945b79cf248") : configmap "kube-rbac-proxy" not found
Oct 30 21:42:24.230908 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:24.230845    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-7t4kc-b65p7" podStartSLOduration=2.2308249 podStartE2EDuration="2.2308249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:42:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:24.113505232 +0000 UTC m=+38.923835254" watchObservedRunningTime="2026-10-30 21:42:24.2308249 +0000 UTC m=+39.041154921"
Oct 30 21:42:24.262504 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:24.262474    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-th574"
Oct 30 21:42:24.655359 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:24.655335    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-th574"]
Oct 30 21:42:24.661684 ip-10-0-130-60 kubenswrapper[2580]: W1030 21:42:24.661643    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2386477_903e_4c6c_b807_3f473a58a6b6.slice/crio-8da43005c1309e0d14d60d540ea37689ef09c76768e5ae580bcfed68f38e70e7 WatchSource:0}: Error finding container 8da43005c1309e0d14d60d540ea37689ef09c76768e5ae580bcfed68f38e70e7: Status 404 returned error can't find the container with id 8da43005c1309e0d14d60d540ea37689ef09c76768e5ae580bcfed68f38e70e7
Oct 30 21:42:24.898641 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:24.898555    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-q4khr" event={"ID":"033356b5-3c05-4f49-ad6a-7c369259970d","Type":"ContainerStarted","Data":"8225a631aca29305c9d7474caa24f61a224b4713f9be0e2bd79976ecc42d2a51"}
Oct 30 21:42:24.900955 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:24.900925    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-rznzd" event={"ID":"879022d4-06b9-44cc-a2a2-dc307b27345e","Type":"ContainerStarted","Data":"ab3c7dac75ae3ce341c2090226f7ce120ec5f8edf58b1c289dfe44c4bd5e66ec"}
Oct 30 21:42:24.902688 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:24.902663    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-th574" event={"ID":"c2386477-903e-4c6c-b807-3f473a58a6b6","Type":"ContainerStarted","Data":"614bb180cbf5031ae47d5cec7f0f9d51cd57e422bef828c188ff3b9a44e6623f"}
Oct 30 21:42:24.902769 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:24.902718    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-th574" event={"ID":"c2386477-903e-4c6c-b807-3f473a58a6b6","Type":"ContainerStarted","Data":"8da43005c1309e0d14d60d540ea37689ef09c76768e5ae580bcfed68f38e70e7"}
Oct 30 21:42:25.010214 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:25.010157    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-q4khr" podStartSLOduration=1.1297054229999999 podStartE2EDuration="3.010138056s" totalImagesPullingTime="1.880432633s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:42:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:25.008566998 +0000 UTC m=+39.818897222" watchObservedRunningTime="2026-10-30 21:42:25.010138056 +0000 UTC m=+39.820468080"
Oct 30 21:42:25.119953 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:25.119886    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-rznzd" podStartSLOduration=1.357738034 podStartE2EDuration="3.119867135s" totalImagesPullingTime="1.762129101s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:42:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:25.117375774 +0000 UTC m=+39.927705801" watchObservedRunningTime="2026-10-30 21:42:25.119867135 +0000 UTC m=+39.930197161"
Oct 30 21:42:25.231786 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:25.231684    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-th574" podStartSLOduration=2.231667134 podStartE2EDuration="2.231667134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:42:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:25.230307251 +0000 UTC m=+40.040637273" watchObservedRunningTime="2026-10-30 21:42:25.231667134 +0000 UTC m=+40.041997158"
Oct 30 21:42:25.841419 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:25.841381    2580 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-sclrt"
Oct 30 21:42:25.841613 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:25.841451    2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-sclrt"
Oct 30 21:42:25.841850 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:25.841836    2580 scope.go:122] "RemoveContainer" containerID="fef0614d3ea43c5da061d6b88ea8aa29855cf27a2684dc399ca63400506200bb"
Oct 30 21:42:25.842037 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:25.842017    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-sclrt_openshift-console-operator(5e74071e-0462-42b5-a05c-d90bb638478a)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-sclrt" podUID="5e74071e-0462-42b5-a05c-d90bb638478a"
Oct 30 21:42:26.213593 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:26.213304    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/74ff988a-c05e-4a3e-adff-5945b79cf248-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mstdj\" (UID: \"74ff988a-c05e-4a3e-adff-5945b79cf248\") " pod="openshift-insights/insights-runtime-extractor-mstdj"
Oct 30 21:42:26.213593 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:26.213360    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/74ff988a-c05e-4a3e-adff-5945b79cf248-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mstdj\" (UID: \"74ff988a-c05e-4a3e-adff-5945b79cf248\") " pod="openshift-insights/insights-runtime-extractor-mstdj"
Oct 30 21:42:26.213593 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:26.213429    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 21:42:26.213593 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:26.213494    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 21:42:26.213593 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:26.213502    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/74ff988a-c05e-4a3e-adff-5945b79cf248-kube-rbac-proxy-cm podName:74ff988a-c05e-4a3e-adff-5945b79cf248 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:30.213482831 +0000 UTC m=+45.023812844 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/74ff988a-c05e-4a3e-adff-5945b79cf248-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mstdj" (UID: "74ff988a-c05e-4a3e-adff-5945b79cf248") : configmap "kube-rbac-proxy" not found
Oct 30 21:42:26.213593 ip-10-0-130-60 kubenswrapper[2580]: E1030 21:42:26.213566    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74ff988a-c05e-4a3e-adff-5945b79cf248-insights-runtime-extractor-tls podName:74ff988a-c05e-4a3e-adff-5945b79cf248 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:30.213547575 +0000 UTC m=+45.023877598 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/74ff988a-c05e-4a3e-adff-5945b79cf248-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mstdj" (UID: "74ff988a-c05e-4a3e-adff-5945b79cf248") : secret "insights-runtime-extractor-tls" not found
Oct 30 21:42:26.313686 ip-10-0-130-60 kubenswrapper[2580]: I1030 21:42:26.313274    2580 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 21:42:26.313442 ip-10-0-130-60 systemd[1]: Stopping Kubernetes Kubelet...
Oct 30 21:42:26.324019 ip-10-0-130-60 systemd[1]: kubelet.service: Deactivated successfully.
Oct 30 21:42:26.324234 ip-10-0-130-60 systemd[1]: Stopped Kubernetes Kubelet.
Oct 30 21:42:26.324458 ip-10-0-130-60 systemd[1]: kubelet.service: Consumed 3.148s CPU time.
Oct 30 21:42:26.334087 ip-10-0-130-60 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 21:42:26.424768 ip-10-0-130-60 kubenswrapper[5061]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 21:42:26.424768 ip-10-0-130-60 kubenswrapper[5061]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 21:42:26.424768 ip-10-0-130-60 kubenswrapper[5061]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 21:42:26.424768 ip-10-0-130-60 kubenswrapper[5061]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 21:42:26.428912 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.428819    5061 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:42:26.428912 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.428912    5061 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:42:26.428912 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.428917    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:42:26.429075 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.428921    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:42:26.429075 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.428929    5061 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:42:26.429075 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.428935    5061 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:42:26.429075 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.428940    5061 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:42:26.429075 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.428945    5061 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:42:26.429075 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.428949    5061 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:42:26.429075 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.428953    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:42:26.429075 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.428957    5061 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:42:26.429075 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.428961    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:42:26.429075 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.428966    5061 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:42:26.429075 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.428970    5061 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:42:26.429075 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.428974    5061 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:42:26.429075 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.428978    5061 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:42:26.429075 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.428982    5061 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:42:26.429075 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.428986    5061 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:42:26.429075 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.428990    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:42:26.429075 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.428994    5061 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:42:26.429075 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.428998    5061 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:42:26.429860 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.429002    5061 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:42:26.429860 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.429006    5061 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:42:26.429860 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.429011    5061 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:42:26.429860 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.429015    5061 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:42:26.429860 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.429019    5061 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:42:26.429860 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.429024    5061 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:42:26.429860 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.429028    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:42:26.429860 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.429032    5061 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:42:26.429860 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.429036    5061 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:42:26.429860 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.429040    5061 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:42:26.429860 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.429051    5061 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:42:26.429860 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.429055    5061 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:42:26.429860 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.429059    5061 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:42:26.429860 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.429064    5061 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:42:26.429860 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.429068    5061 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:42:26.429860 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.429072    5061 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:42:26.429860 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.429076    5061 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:42:26.429860 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.429080    5061 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:42:26.429860 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.429084    5061 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:42:26.430634 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.429088    5061 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:42:26.430634 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.429093    5061 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:42:26.430634 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.429097    5061 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:42:26.430634 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.429102    5061 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:42:26.430634 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.429106    5061 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:42:26.430634 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.429110    5061 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:42:26.430634 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.429114    5061 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:42:26.430634 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.429118    5061 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:42:26.430634 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.429123    5061 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:42:26.430634 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.429127    5061 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:42:26.430634 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.429131    5061 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:42:26.430634 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.429135    5061 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:42:26.430634 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.429139    5061 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:42:26.430634 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.429143    5061 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:42:26.430634 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.429147    5061 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:42:26.430634 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.429152    5061 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:42:26.430634 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.429156    5061 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:42:26.430634 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.429160    5061 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:42:26.430634 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.429164    5061 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:42:26.430634 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.429168    5061 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:42:26.431427 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.429172    5061 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:42:26.431427 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.429177    5061 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:42:26.431427 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.429181    5061 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:42:26.431427 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.429185    5061 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:42:26.431427 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.429190    5061 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:42:26.431427 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.429194    5061 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:42:26.431427 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.429198    5061 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:42:26.431427 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.429206    5061 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:42:26.431427 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.429210    5061 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:42:26.431427 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.429214    5061 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:42:26.431427 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.429218    5061 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:42:26.431427 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.429223    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:42:26.431427 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.429227    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:42:26.431427 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.429231    5061 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:42:26.431427 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.429235    5061 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:42:26.431427 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.429239    5061 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:42:26.431427 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.429244    5061 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:42:26.431427 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.429248    5061 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:42:26.431427 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.429252    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:42:26.431427 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.429256    5061 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:42:26.432265 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.429260    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:42:26.432265 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.429264    5061 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:42:26.432265 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.429268    5061 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:42:26.432265 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.429272    5061 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:42:26.432265 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.429277    5061 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:42:26.432265 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.429281    5061 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:42:26.432265 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.429285    5061 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:42:26.432265 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.429289    5061 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:42:26.432265 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.429293    5061 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:42:26.432265 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.429298    5061 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:42:26.432265 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.429302    5061 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:42:26.432265 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.429306    5061 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:42:26.432265 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.429310    5061 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:42:26.432265 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.429314    5061 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:42:26.432265 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.429318    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:42:26.432265 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.429323    5061 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:42:26.432265 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.429327    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:42:26.432265 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.429331    5061 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:42:26.432265 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.429335    5061 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:42:26.433028 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.429341    5061 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:42:26.433028 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.429346    5061 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:42:26.433028 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.429350    5061 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:42:26.433028 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.429356    5061 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:42:26.433028 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.429363    5061 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:42:26.433028 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.429367    5061 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:42:26.433028 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.429371    5061 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:42:26.433028 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.430148    5061 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:42:26.433028 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.430155    5061 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:42:26.433028 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.430158    5061 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:42:26.433028 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.430161    5061 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:42:26.433028 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.430164    5061 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:42:26.433028 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.430168    5061 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:42:26.433028 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.430170    5061 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:42:26.433028 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.430173    5061 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:42:26.433028 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.430176    5061 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:42:26.433028 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.430179    5061 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:42:26.433028 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.430182    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:42:26.433028 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.430185    5061 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:42:26.433831 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.430187    5061 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:42:26.433831 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.430190    5061 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:42:26.433831 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.430192    5061 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:42:26.433831 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.430195    5061 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:42:26.433831 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.430198    5061 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:42:26.433831 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.430201    5061 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:42:26.433831 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.430204    5061 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:42:26.433831 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.430206    5061 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:42:26.433831 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.430209    5061 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:42:26.433831 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.430211    5061 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:42:26.433831 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.430214    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:42:26.433831 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.430216    5061 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:42:26.433831 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.430219    5061 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:42:26.433831 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.430224    5061 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:42:26.433831 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.430227    5061 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:42:26.433831 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.430230    5061 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:42:26.433831 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.430232    5061 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:42:26.433831 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.430235    5061 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:42:26.433831 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.430237    5061 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:42:26.434620 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.430240    5061 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:42:26.434620 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.430242    5061 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:42:26.434620 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.430245    5061 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:42:26.434620 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.430247    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:42:26.434620 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.430250    5061 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:42:26.434620 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.430253    5061 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:42:26.434620 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.430255    5061 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:42:26.434620 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.430260    5061 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:42:26.434620 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.430265    5061 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:42:26.434620 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.430269    5061 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:42:26.434620 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.430272    5061 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:42:26.434620 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.430283    5061 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:42:26.434620 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.430288    5061 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:42:26.434620 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.430292    5061 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:42:26.434620 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.430295    5061 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:42:26.434620 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.430297    5061 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:42:26.434620 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.430300    5061 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:42:26.434620 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.430303    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:42:26.434620 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.430305    5061 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:42:26.434620 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.430308    5061 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:42:26.435402 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.430311    5061 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:42:26.435402 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.430314    5061 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:42:26.435402 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.430316    5061 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:42:26.435402 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.430319    5061 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:42:26.435402 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.430322    5061 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:42:26.435402 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.430324    5061 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:42:26.435402 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.430327    5061 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:42:26.435402 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.430330    5061 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:42:26.435402 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.430332    5061 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:42:26.435402 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.430335    5061 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:42:26.435402 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.430337    5061 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:42:26.435402 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.430340    5061 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:42:26.435402 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.430343    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:42:26.435402 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.430345    5061 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:42:26.435402 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.430348    5061 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:42:26.435402 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.430350    5061 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:42:26.435402 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.430353    5061 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:42:26.435402 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.430356    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:42:26.435402 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.430358    5061 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:42:26.435402 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.430361    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:42:26.436274 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.430364    5061 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:42:26.436274 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.430366    5061 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:42:26.436274 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.430369    5061 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:42:26.436274 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.430372    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:42:26.436274 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.430374    5061 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:42:26.436274 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.430377    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:42:26.436274 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.430379    5061 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:42:26.436274 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.430382    5061 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:42:26.436274 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.430385    5061 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:42:26.436274 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.430387    5061 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:42:26.436274 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.430390    5061 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:42:26.436274 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.430392    5061 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:42:26.436274 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.430395    5061 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:42:26.436274 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.430398    5061 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:42:26.436274 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.430401    5061 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:42:26.436274 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.430406    5061 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:42:26.436274 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.430408    5061 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:42:26.436274 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.430411    5061 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:42:26.436274 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.430414    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:42:26.436274 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.430416    5061 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:42:26.437147 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.430419    5061 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:42:26.437147 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.430421    5061 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:42:26.437147 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.430425    5061 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:42:26.437147 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.430428    5061 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:42:26.437147 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.430430    5061 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:42:26.437147 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.430433    5061 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:42:26.437147 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.430435    5061 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:42:26.437147 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.430438    5061 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:42:26.437147 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.430440    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:42:26.437147 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.430443    5061 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:42:26.437147 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.430446    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:42:26.437147 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.430449    5061 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:42:26.437147 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.430451    5061 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:42:26.437147 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.430454    5061 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:42:26.437147 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.430457    5061 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:42:26.437147 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.430974    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:42:26.437147 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.430981    5061 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:42:26.437147 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.430984    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:42:26.437147 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.430987    5061 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:42:26.437898 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.430990    5061 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:42:26.437898 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.430993    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:42:26.437898 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.430996    5061 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:42:26.437898 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.430999    5061 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:42:26.437898 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.431002    5061 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:42:26.437898 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.431005    5061 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:42:26.437898 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.431007    5061 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:42:26.437898 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.431010    5061 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:42:26.437898 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.431016    5061 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:42:26.437898 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.431019    5061 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:42:26.437898 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.431021    5061 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:42:26.437898 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.431024    5061 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:42:26.437898 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.431027    5061 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:42:26.437898 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.431029    5061 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:42:26.437898 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.431032    5061 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:42:26.437898 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.431035    5061 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:42:26.437898 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.431037    5061 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:42:26.437898 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.431040    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:42:26.437898 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.431042    5061 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:42:26.437898 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.431045    5061 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:42:26.438757 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.431047    5061 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:42:26.438757 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.431050    5061 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:42:26.438757 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.431052    5061 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:42:26.438757 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.431055    5061 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:42:26.438757 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.431057    5061 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:42:26.438757 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.431060    5061 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:42:26.438757 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.431063    5061 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:42:26.438757 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.431066    5061 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:42:26.438757 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.431068    5061 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:42:26.438757 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.431071    5061 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:42:26.438757 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.431074    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:42:26.438757 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.431076    5061 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:42:26.438757 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.431079    5061 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:42:26.438757 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.431081    5061 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:42:26.438757 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.431083    5061 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:42:26.438757 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.431086    5061 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:42:26.438757 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.431089    5061 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:42:26.438757 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.431091    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:42:26.438757 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.431094    5061 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:42:26.439614 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.431096    5061 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:42:26.439614 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.431100    5061 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:42:26.439614 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.431103    5061 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:42:26.439614 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.431106    5061 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:42:26.439614 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.431108    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:42:26.439614 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.431111    5061 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:42:26.439614 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.431113    5061 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:42:26.439614 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.431116    5061 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:42:26.439614 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.431118    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:42:26.439614 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.431121    5061 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:42:26.439614 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.431123    5061 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:42:26.439614 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.431126    5061 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:42:26.439614 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.431128    5061 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:42:26.439614 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.431131    5061 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:42:26.439614 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.431133    5061 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:42:26.439614 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.431136    5061 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:42:26.439614 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.431138    5061 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:42:26.439614 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.431141    5061 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:42:26.439614 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.431144    5061 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:42:26.439614 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.431146    5061 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:42:26.440397 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.431149    5061 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:42:26.440397 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.431152    5061 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:42:26.440397 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.431154    5061 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:42:26.440397 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.431157    5061 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:42:26.440397 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.431160    5061 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:42:26.440397 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.431162    5061 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:42:26.440397 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.431165    5061 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:42:26.440397 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.431167    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:42:26.440397 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.431170    5061 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:42:26.440397 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.431173    5061 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:42:26.440397 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.431175    5061 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:42:26.440397 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.431179    5061 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:42:26.440397 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.431183    5061 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:42:26.440397 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.431188    5061 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:42:26.440397 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.431191    5061 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:42:26.440397 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.431194    5061 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:42:26.440397 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.431196    5061 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:42:26.440397 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.431199    5061 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:42:26.440397 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.431202    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:42:26.441149 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.431206    5061 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:42:26.441149 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.431209    5061 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:42:26.441149 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.431212    5061 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:42:26.441149 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.431215    5061 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:42:26.441149 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.431218    5061 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:42:26.441149 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.431221    5061 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:42:26.441149 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.431224    5061 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:42:26.441149 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.431226    5061 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:42:26.441149 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.431229    5061 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:42:26.441149 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.431231    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:42:26.441149 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.431234    5061 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:42:26.441149 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.431237    5061 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:42:26.441149 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.431239    5061 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:42:26.441149 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.431242    5061 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:42:26.441149 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.431245    5061 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:42:26.441149 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.431248    5061 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:42:26.441149 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.431251    5061 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:42:26.441149 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.431253    5061 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:42:26.441149 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.431255    5061 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:42:26.441149 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.431258    5061 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:42:26.441904 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.431260    5061 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:42:26.441904 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.431263    5061 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:42:26.441904 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.431265    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:42:26.441904 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.431268    5061 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:42:26.441904 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431358    5061 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 21:42:26.441904 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431369    5061 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 21:42:26.441904 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431380    5061 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 21:42:26.441904 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431386    5061 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 21:42:26.441904 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431393    5061 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 21:42:26.441904 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431400    5061 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 21:42:26.441904 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431405    5061 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 21:42:26.441904 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431409    5061 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 21:42:26.441904 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431413    5061 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 21:42:26.441904 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431416    5061 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 21:42:26.441904 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431419    5061 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 21:42:26.441904 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431423    5061 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 21:42:26.441904 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431426    5061 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 21:42:26.441904 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431429    5061 flags.go:64] FLAG: --cgroup-root=""
Oct 30 21:42:26.441904 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431432    5061 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 21:42:26.441904 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431435    5061 flags.go:64] FLAG: --client-ca-file=""
Oct 30 21:42:26.441904 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431438    5061 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 21:42:26.441904 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431441    5061 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 21:42:26.441904 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431445    5061 flags.go:64] FLAG: --cluster-domain=""
Oct 30 21:42:26.442580 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431448    5061 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 21:42:26.442580 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431451    5061 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 21:42:26.442580 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431454    5061 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 21:42:26.442580 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431458    5061 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 21:42:26.442580 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431463    5061 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 21:42:26.442580 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431466    5061 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 21:42:26.442580 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431469    5061 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 21:42:26.442580 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431472    5061 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 21:42:26.442580 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431475    5061 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 21:42:26.442580 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431478    5061 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 21:42:26.442580 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431481    5061 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 21:42:26.442580 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431484    5061 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 21:42:26.442580 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431487    5061 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 21:42:26.442580 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431491    5061 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 21:42:26.442580 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431495    5061 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 21:42:26.442580 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431499    5061 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 21:42:26.442580 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431502    5061 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 21:42:26.442580 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431505    5061 flags.go:64] FLAG: --enable-server="true"
Oct 30 21:42:26.442580 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431509    5061 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 21:42:26.442580 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431514    5061 flags.go:64] FLAG: --event-burst="100"
Oct 30 21:42:26.442580 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431517    5061 flags.go:64] FLAG: --event-qps="50"
Oct 30 21:42:26.442580 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431521    5061 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 21:42:26.442580 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431536    5061 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 21:42:26.442580 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431539    5061 flags.go:64] FLAG: --eviction-hard=""
Oct 30 21:42:26.443406 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431544    5061 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 21:42:26.443406 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431547    5061 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 21:42:26.443406 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431550    5061 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 21:42:26.443406 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431553    5061 flags.go:64] FLAG: --eviction-soft=""
Oct 30 21:42:26.443406 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431556    5061 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 21:42:26.443406 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431559    5061 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 21:42:26.443406 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431562    5061 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 21:42:26.443406 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431565    5061 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 21:42:26.443406 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431568    5061 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 21:42:26.443406 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431571    5061 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 21:42:26.443406 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431574    5061 flags.go:64] FLAG: --feature-gates=""
Oct 30 21:42:26.443406 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431578    5061 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 21:42:26.443406 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431581    5061 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 21:42:26.443406 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431585    5061 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 21:42:26.443406 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431588    5061 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 21:42:26.443406 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431591    5061 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 21:42:26.443406 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431594    5061 flags.go:64] FLAG: --help="false"
Oct 30 21:42:26.443406 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431597    5061 flags.go:64] FLAG: --hostname-override="ip-10-0-130-60.us-west-2.compute.internal"
Oct 30 21:42:26.443406 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431600    5061 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 21:42:26.443406 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431604    5061 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 21:42:26.443406 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431606    5061 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 21:42:26.443406 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431610    5061 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 21:42:26.443406 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431614    5061 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 21:42:26.444349 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431618    5061 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 21:42:26.444349 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431621    5061 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 21:42:26.444349 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431624    5061 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 21:42:26.444349 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431628    5061 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 21:42:26.444349 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431631    5061 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 21:42:26.444349 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431634    5061 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 21:42:26.444349 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431637    5061 flags.go:64] FLAG: --kube-reserved=""
Oct 30 21:42:26.444349 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431640    5061 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 21:42:26.444349 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431643    5061 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 21:42:26.444349 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431646    5061 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 21:42:26.444349 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431649    5061 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 21:42:26.444349 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431652    5061 flags.go:64] FLAG: --lock-file=""
Oct 30 21:42:26.444349 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431655    5061 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 21:42:26.444349 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431658    5061 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 21:42:26.444349 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431661    5061 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 21:42:26.444349 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431667    5061 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 21:42:26.444349 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431670    5061 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 21:42:26.444349 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431673    5061 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 21:42:26.444349 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431676    5061 flags.go:64] FLAG: --logging-format="text"
Oct 30 21:42:26.444349 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431679    5061 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 21:42:26.444349 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431683    5061 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 21:42:26.444349 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431685    5061 flags.go:64] FLAG: --manifest-url=""
Oct 30 21:42:26.444349 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431688    5061 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 21:42:26.444349 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431693    5061 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 21:42:26.444349 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431696    5061 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 21:42:26.445362 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431700    5061 flags.go:64] FLAG: --max-pods="110"
Oct 30 21:42:26.445362 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431703    5061 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 21:42:26.445362 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431706    5061 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 21:42:26.445362 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431709    5061 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 21:42:26.445362 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431712    5061 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 21:42:26.445362 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431715    5061 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 21:42:26.445362 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431718    5061 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 21:42:26.445362 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431722    5061 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 21:42:26.445362 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431729    5061 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 21:42:26.445362 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431732    5061 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 21:42:26.445362 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431737    5061 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 21:42:26.445362 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431740    5061 flags.go:64] FLAG: --pod-cidr=""
Oct 30 21:42:26.445362 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431743    5061 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 21:42:26.445362 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431745    5061 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 21:42:26.445362 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431748    5061 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 21:42:26.445362 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431751    5061 flags.go:64] FLAG: --port="10250"
Oct 30 21:42:26.445362 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431754    5061 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 21:42:26.445362 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431757    5061 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-08205b8e8f123ed6b"
Oct 30 21:42:26.445362 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431761    5061 flags.go:64] FLAG: --qos-reserved=""
Oct 30 21:42:26.445362 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431764    5061 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 21:42:26.445362 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431767    5061 flags.go:64] FLAG: --register-node="true"
Oct 30 21:42:26.445362 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431770    5061 flags.go:64] FLAG: --register-with-taints=""
Oct 30 21:42:26.445362 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431774    5061 flags.go:64] FLAG: --registry-burst="10"
Oct 30 21:42:26.445362 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431777    5061 flags.go:64] FLAG: --registry-qps="5"
Oct 30 21:42:26.445362 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431779    5061 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 21:42:26.445362 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431782    5061 flags.go:64] FLAG: --reserved-memory=""
Oct 30 21:42:26.446468 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431786    5061 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 21:42:26.446468 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431789    5061 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 21:42:26.446468 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431792    5061 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 21:42:26.446468 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431795    5061 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 21:42:26.446468 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431798    5061 flags.go:64] FLAG: --runonce="false"
Oct 30 21:42:26.446468 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431801    5061 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 21:42:26.446468 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431804    5061 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 21:42:26.446468 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431807    5061 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 21:42:26.446468 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431810    5061 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 21:42:26.446468 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431813    5061 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 21:42:26.446468 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431816    5061 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 21:42:26.446468 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431819    5061 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 21:42:26.446468 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431823    5061 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 21:42:26.446468 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431827    5061 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 21:42:26.446468 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431830    5061 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 21:42:26.446468 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431833    5061 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 21:42:26.446468 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431837    5061 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 21:42:26.446468 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431840    5061 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 21:42:26.446468 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431843    5061 flags.go:64] FLAG: --system-cgroups=""
Oct 30 21:42:26.446468 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431846    5061 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 21:42:26.446468 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431851    5061 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 21:42:26.446468 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431854    5061 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 21:42:26.446468 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431857    5061 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 21:42:26.446468 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431860    5061 flags.go:64] FLAG: --tls-min-version=""
Oct 30 21:42:26.446468 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431863    5061 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 21:42:26.447506 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431866    5061 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 21:42:26.447506 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431869    5061 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 21:42:26.447506 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431872    5061 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 21:42:26.447506 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431875    5061 flags.go:64] FLAG: --v="2"
Oct 30 21:42:26.447506 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431879    5061 flags.go:64] FLAG: --version="false"
Oct 30 21:42:26.447506 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431883    5061 flags.go:64] FLAG: --vmodule=""
Oct 30 21:42:26.447506 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431886    5061 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 21:42:26.447506 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.431890    5061 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 21:42:26.447506 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432012    5061 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:42:26.447506 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432016    5061 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:42:26.447506 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432019    5061 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:42:26.447506 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432022    5061 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:42:26.447506 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432025    5061 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:42:26.447506 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432028    5061 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:42:26.447506 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432031    5061 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:42:26.447506 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432033    5061 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:42:26.447506 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432036    5061 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:42:26.447506 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432039    5061 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:42:26.447506 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432042    5061 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:42:26.447506 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432044    5061 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:42:26.447506 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432048    5061 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:42:26.448347 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432051    5061 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:42:26.448347 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432054    5061 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:42:26.448347 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432058    5061 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:42:26.448347 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432060    5061 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:42:26.448347 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432063    5061 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:42:26.448347 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432065    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:42:26.448347 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432068    5061 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:42:26.448347 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432071    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:42:26.448347 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432073    5061 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:42:26.448347 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432076    5061 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:42:26.448347 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432079    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:42:26.448347 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432081    5061 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:42:26.448347 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432084    5061 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:42:26.448347 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432086    5061 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:42:26.448347 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432089    5061 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:42:26.448347 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432092    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:42:26.448347 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432095    5061 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:42:26.448347 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432098    5061 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:42:26.448347 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432100    5061 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:42:26.448347 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432103    5061 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:42:26.449163 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432106    5061 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:42:26.449163 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432109    5061 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:42:26.449163 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432112    5061 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:42:26.449163 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432116    5061 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:42:26.449163 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432120    5061 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:42:26.449163 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432125    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:42:26.449163 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432128    5061 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:42:26.449163 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432132    5061 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:42:26.449163 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432135    5061 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:42:26.449163 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432137    5061 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:42:26.449163 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432140    5061 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:42:26.449163 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432144    5061 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:42:26.449163 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432147    5061 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:42:26.449163 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432149    5061 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:42:26.449163 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432153    5061 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:42:26.449163 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432156    5061 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:42:26.449163 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432159    5061 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:42:26.449163 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432162    5061 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:42:26.449163 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432165    5061 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:42:26.449950 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432167    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:42:26.449950 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432170    5061 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:42:26.449950 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432173    5061 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:42:26.449950 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432176    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:42:26.449950 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432178    5061 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:42:26.449950 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432181    5061 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:42:26.449950 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432184    5061 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:42:26.449950 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432186    5061 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:42:26.449950 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432189    5061 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:42:26.449950 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432192    5061 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:42:26.449950 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432194    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:42:26.449950 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432197    5061 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:42:26.449950 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432199    5061 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:42:26.449950 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432202    5061 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:42:26.449950 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432205    5061 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:42:26.449950 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432207    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:42:26.449950 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432210    5061 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:42:26.449950 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432213    5061 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:42:26.449950 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432215    5061 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:42:26.449950 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432218    5061 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:42:26.450762 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432221    5061 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:42:26.450762 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432223    5061 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:42:26.450762 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432226    5061 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:42:26.450762 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432228    5061 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:42:26.450762 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432232    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:42:26.450762 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432235    5061 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:42:26.450762 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432238    5061 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:42:26.450762 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432241    5061 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:42:26.450762 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432244    5061 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:42:26.450762 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432247    5061 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:42:26.450762 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432250    5061 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:42:26.450762 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432252    5061 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:42:26.450762 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432255    5061 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:42:26.450762 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432258    5061 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:42:26.450762 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432260    5061 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:42:26.450762 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432263    5061 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:42:26.450762 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432265    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:42:26.450762 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432268    5061 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:42:26.450762 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432271    5061 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:42:26.451516 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432274    5061 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:42:26.451516 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432277    5061 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:42:26.451516 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432280    5061 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:42:26.451516 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432283    5061 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:42:26.451516 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432285    5061 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:42:26.451516 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432288    5061 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:42:26.451516 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432291    5061 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:42:26.451516 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432293    5061 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:42:26.451516 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432296    5061 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:42:26.451516 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432299    5061 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:42:26.451516 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432301    5061 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:42:26.451516 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432304    5061 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:42:26.451516 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432307    5061 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:42:26.451516 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432310    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:42:26.451516 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432313    5061 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:42:26.451516 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.432338    5061 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 21:42:26.451516 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432378    5061 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:42:26.452201 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432382    5061 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:42:26.452201 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432386    5061 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:42:26.452201 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432389    5061 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:42:26.452201 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432393    5061 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:42:26.452201 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432396    5061 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:42:26.452201 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432398    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:42:26.452201 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432401    5061 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:42:26.452201 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432404    5061 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:42:26.452201 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432406    5061 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:42:26.452201 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432409    5061 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:42:26.452201 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432412    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:42:26.452201 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432414    5061 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:42:26.452201 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432417    5061 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:42:26.452201 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432420    5061 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:42:26.452201 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432423    5061 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:42:26.452201 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432425    5061 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:42:26.452201 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432428    5061 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:42:26.452201 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432431    5061 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:42:26.452201 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432433    5061 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:42:26.452201 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432436    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:42:26.453007 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432439    5061 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:42:26.453007 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432441    5061 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:42:26.453007 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432444    5061 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:42:26.453007 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432446    5061 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:42:26.453007 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432449    5061 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:42:26.453007 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432452    5061 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:42:26.453007 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432454    5061 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:42:26.453007 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432457    5061 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:42:26.453007 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432459    5061 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:42:26.453007 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432463    5061 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:42:26.453007 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432465    5061 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:42:26.453007 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432468    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:42:26.453007 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432470    5061 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:42:26.453007 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432474    5061 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:42:26.453007 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432477    5061 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:42:26.453007 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432481    5061 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:42:26.453007 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432483    5061 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:42:26.453007 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432486    5061 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:42:26.453007 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432489    5061 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:42:26.453784 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432491    5061 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:42:26.453784 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432494    5061 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:42:26.453784 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432497    5061 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:42:26.453784 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432499    5061 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:42:26.453784 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432503    5061 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:42:26.453784 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432505    5061 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:42:26.453784 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432508    5061 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:42:26.453784 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432510    5061 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:42:26.453784 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432513    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:42:26.453784 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432516    5061 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:42:26.453784 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432518    5061 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:42:26.453784 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432521    5061 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:42:26.453784 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432534    5061 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:42:26.453784 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432537    5061 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:42:26.453784 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432540    5061 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:42:26.453784 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432543    5061 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:42:26.453784 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432545    5061 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:42:26.453784 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432548    5061 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:42:26.453784 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432551    5061 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:42:26.453784 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432554    5061 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:42:26.454668 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432556    5061 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:42:26.454668 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432559    5061 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:42:26.454668 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432562    5061 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:42:26.454668 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432565    5061 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:42:26.454668 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432567    5061 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:42:26.454668 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432570    5061 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:42:26.454668 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432576    5061 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:42:26.454668 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432579    5061 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:42:26.454668 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432584    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:42:26.454668 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432586    5061 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:42:26.454668 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432589    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:42:26.454668 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432592    5061 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:42:26.454668 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432594    5061 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:42:26.454668 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432597    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:42:26.454668 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432599    5061 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:42:26.454668 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432602    5061 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:42:26.454668 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432605    5061 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:42:26.454668 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432608    5061 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:42:26.454668 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432610    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:42:26.454668 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432613    5061 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:42:26.455507 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432616    5061 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:42:26.455507 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432619    5061 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:42:26.455507 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432621    5061 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:42:26.455507 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432624    5061 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:42:26.455507 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432628    5061 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:42:26.455507 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432631    5061 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:42:26.455507 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432636    5061 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:42:26.455507 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432640    5061 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:42:26.455507 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432643    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:42:26.455507 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432646    5061 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:42:26.455507 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432649    5061 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:42:26.455507 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432652    5061 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:42:26.455507 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432655    5061 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:42:26.455507 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432658    5061 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:42:26.455507 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432661    5061 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:42:26.455507 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432663    5061 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:42:26.455507 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432666    5061 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:42:26.455507 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432669    5061 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:42:26.455507 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432673    5061 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:42:26.456284 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432677    5061 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:42:26.456284 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432681    5061 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:42:26.456284 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432683    5061 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:42:26.456284 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432686    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:42:26.456284 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432689    5061 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:42:26.456284 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432691    5061 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:42:26.456284 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.432694    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:42:26.456284 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.436688    5061 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 21:42:26.456284 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.436704    5061 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 21:42:26.456284 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.436718    5061 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 21:42:26.456284 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.436723    5061 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 21:42:26.456284 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.436777    5061 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:42:26.456284 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.436782    5061 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:42:26.456284 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.436785    5061 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:42:26.456284 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.436788    5061 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:42:26.456284 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.436791    5061 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:42:26.456284 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.436794    5061 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:42:26.456284 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.436797    5061 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:42:26.456284 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.436800    5061 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:42:26.456284 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.436802    5061 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:42:26.457078 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.436805    5061 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:42:26.457078 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.436808    5061 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:42:26.457078 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.436811    5061 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:42:26.457078 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.436814    5061 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:42:26.457078 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.436817    5061 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:42:26.457078 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.436819    5061 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:42:26.457078 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.436822    5061 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:42:26.457078 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.436825    5061 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:42:26.457078 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.436827    5061 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:42:26.457078 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.436830    5061 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:42:26.457078 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.436833    5061 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:42:26.457078 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.436836    5061 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:42:26.457078 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.436838    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:42:26.457078 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.436841    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:42:26.457078 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.436844    5061 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:42:26.457078 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.436848    5061 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:42:26.457078 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.436850    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:42:26.457078 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.436853    5061 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:42:26.457078 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.436856    5061 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:42:26.457078 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.436858    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:42:26.457876 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.436861    5061 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:42:26.457876 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.436864    5061 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:42:26.457876 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.436867    5061 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:42:26.457876 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.436870    5061 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:42:26.457876 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.436872    5061 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:42:26.457876 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.436875    5061 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:42:26.457876 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.436877    5061 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:42:26.457876 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.436881    5061 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:42:26.457876 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.436884    5061 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:42:26.457876 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.436886    5061 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:42:26.457876 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.436889    5061 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:42:26.457876 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.436892    5061 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:42:26.457876 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.436895    5061 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:42:26.457876 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.436897    5061 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:42:26.457876 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.436900    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:42:26.457876 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.436903    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:42:26.457876 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.436906    5061 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:42:26.457876 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.436908    5061 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:42:26.457876 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.436911    5061 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:42:26.458684 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.436914    5061 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:42:26.458684 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.436916    5061 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:42:26.458684 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.436919    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:42:26.458684 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.436922    5061 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:42:26.458684 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.436925    5061 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:42:26.458684 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.436927    5061 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:42:26.458684 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.436930    5061 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:42:26.458684 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.436933    5061 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:42:26.458684 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.436935    5061 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:42:26.458684 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.436938    5061 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:42:26.458684 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.436941    5061 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:42:26.458684 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.436943    5061 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:42:26.458684 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.436946    5061 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:42:26.458684 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.436949    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:42:26.458684 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.436951    5061 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:42:26.458684 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.436954    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:42:26.458684 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.436957    5061 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:42:26.458684 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.436960    5061 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:42:26.458684 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.436962    5061 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:42:26.458684 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.436965    5061 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:42:26.459482 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.436967    5061 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:42:26.459482 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.436970    5061 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:42:26.459482 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.436972    5061 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:42:26.459482 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.436976    5061 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:42:26.459482 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.436979    5061 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:42:26.459482 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.436984    5061 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:42:26.459482 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.436987    5061 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:42:26.459482 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.436990    5061 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:42:26.459482 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.436994    5061 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:42:26.459482 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.436996    5061 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:42:26.459482 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.436999    5061 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:42:26.459482 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437002    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:42:26.459482 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437004    5061 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:42:26.459482 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437007    5061 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:42:26.459482 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437009    5061 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:42:26.459482 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437012    5061 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:42:26.459482 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437015    5061 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:42:26.459482 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437018    5061 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:42:26.459482 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437020    5061 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:42:26.460247 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437023    5061 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:42:26.460247 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437025    5061 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:42:26.460247 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437028    5061 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:42:26.460247 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437030    5061 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:42:26.460247 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437033    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:42:26.460247 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437035    5061 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:42:26.460247 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437038    5061 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:42:26.460247 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437040    5061 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:42:26.460247 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437049    5061 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:42:26.460247 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437052    5061 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:42:26.460247 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437054    5061 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:42:26.460247 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437057    5061 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:42:26.460247 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437060    5061 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:42:26.460247 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437063    5061 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:42:26.460247 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437065    5061 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:42:26.460247 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437068    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:42:26.460247 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437070    5061 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:42:26.460247 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437073    5061 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:42:26.460247 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437075    5061 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:42:26.461018 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.437102    5061 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 21:42:26.461018 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437276    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:42:26.461018 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437281    5061 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:42:26.461018 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437284    5061 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:42:26.461018 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437287    5061 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:42:26.461018 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437290    5061 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:42:26.461018 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437292    5061 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:42:26.461018 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437295    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:42:26.461018 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437297    5061 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:42:26.461018 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437300    5061 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:42:26.461018 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437303    5061 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:42:26.461018 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437305    5061 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:42:26.461018 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437308    5061 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:42:26.461018 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437311    5061 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:42:26.461018 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437313    5061 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:42:26.461018 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437316    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:42:26.461657 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437318    5061 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:42:26.461657 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437321    5061 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:42:26.461657 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437323    5061 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:42:26.461657 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437326    5061 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:42:26.461657 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437328    5061 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:42:26.461657 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437331    5061 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:42:26.461657 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437339    5061 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:42:26.461657 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437342    5061 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:42:26.461657 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437344    5061 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:42:26.461657 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437347    5061 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:42:26.461657 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437350    5061 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:42:26.461657 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437352    5061 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:42:26.461657 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437355    5061 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:42:26.461657 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437357    5061 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:42:26.461657 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437360    5061 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:42:26.461657 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437362    5061 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:42:26.461657 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437365    5061 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:42:26.461657 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437367    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:42:26.461657 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437370    5061 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:42:26.462432 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437372    5061 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:42:26.462432 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437375    5061 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:42:26.462432 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437379    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:42:26.462432 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437382    5061 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:42:26.462432 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437384    5061 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:42:26.462432 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437387    5061 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:42:26.462432 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437389    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:42:26.462432 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437392    5061 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:42:26.462432 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437395    5061 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:42:26.462432 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437397    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:42:26.462432 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437400    5061 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:42:26.462432 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437402    5061 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:42:26.462432 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437405    5061 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:42:26.462432 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437408    5061 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:42:26.462432 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437410    5061 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:42:26.462432 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437413    5061 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:42:26.462432 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437419    5061 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:42:26.462432 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437422    5061 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:42:26.462432 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437424    5061 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:42:26.463328 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437427    5061 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:42:26.463328 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437436    5061 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:42:26.463328 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437439    5061 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:42:26.463328 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437442    5061 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:42:26.463328 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437445    5061 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:42:26.463328 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437447    5061 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:42:26.463328 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437450    5061 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:42:26.463328 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437452    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:42:26.463328 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437455    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:42:26.463328 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437457    5061 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:42:26.463328 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437460    5061 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:42:26.463328 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437463    5061 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:42:26.463328 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437465    5061 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:42:26.463328 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437468    5061 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:42:26.463328 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437471    5061 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:42:26.463328 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437475    5061 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:42:26.463328 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437477    5061 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:42:26.463328 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437480    5061 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:42:26.463328 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437482    5061 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:42:26.463328 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437486    5061 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:42:26.464370 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437489    5061 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:42:26.464370 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437492    5061 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:42:26.464370 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437501    5061 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:42:26.464370 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437504    5061 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:42:26.464370 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437506    5061 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:42:26.464370 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437509    5061 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:42:26.464370 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437511    5061 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:42:26.464370 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437514    5061 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:42:26.464370 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437517    5061 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:42:26.464370 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437521    5061 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:42:26.464370 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437538    5061 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:42:26.464370 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437542    5061 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:42:26.464370 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437546    5061 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:42:26.464370 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437551    5061 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:42:26.464370 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437561    5061 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:42:26.464370 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437565    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:42:26.464370 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437568    5061 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:42:26.464370 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437571    5061 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:42:26.464370 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437573    5061 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:42:26.465171 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437576    5061 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:42:26.465171 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437578    5061 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:42:26.465171 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437581    5061 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:42:26.465171 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437584    5061 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:42:26.465171 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437586    5061 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:42:26.465171 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437589    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:42:26.465171 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437591    5061 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:42:26.465171 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437594    5061 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:42:26.465171 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437598    5061 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:42:26.465171 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437601    5061 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:42:26.465171 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437603    5061 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:42:26.465171 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437606    5061 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:42:26.465171 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437609    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:42:26.465171 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437611    5061 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:42:26.465171 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.437635    5061 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 21:42:26.465171 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437674    5061 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:42:26.465840 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437678    5061 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:42:26.465840 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437681    5061 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:42:26.465840 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437684    5061 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:42:26.465840 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437686    5061 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:42:26.465840 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437689    5061 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:42:26.465840 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437691    5061 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:42:26.465840 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437694    5061 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:42:26.465840 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437698    5061 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:42:26.465840 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437701    5061 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:42:26.465840 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437703    5061 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:42:26.465840 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437706    5061 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:42:26.465840 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437709    5061 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:42:26.465840 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437723    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:42:26.465840 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437726    5061 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:42:26.465840 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437729    5061 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:42:26.465840 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437731    5061 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:42:26.465840 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437734    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:42:26.465840 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437736    5061 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:42:26.465840 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437739    5061 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:42:26.465840 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437742    5061 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:42:26.466703 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437744    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:42:26.466703 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437747    5061 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:42:26.466703 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437750    5061 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:42:26.466703 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437753    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:42:26.466703 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437757    5061 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:42:26.466703 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437759    5061 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:42:26.466703 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437763    5061 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:42:26.466703 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437767    5061 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:42:26.466703 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437772    5061 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:42:26.466703 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437776    5061 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:42:26.466703 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437779    5061 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:42:26.466703 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437784    5061 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:42:26.466703 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437788    5061 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:42:26.466703 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437792    5061 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:42:26.466703 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437796    5061 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:42:26.466703 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437800    5061 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:42:26.466703 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437804    5061 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:42:26.466703 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437808    5061 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:42:26.466703 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437810    5061 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:42:26.467477 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437814    5061 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:42:26.467477 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437817    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:42:26.467477 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437820    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:42:26.467477 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437822    5061 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:42:26.467477 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437825    5061 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:42:26.467477 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437828    5061 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:42:26.467477 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437836    5061 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:42:26.467477 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437839    5061 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:42:26.467477 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437842    5061 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:42:26.467477 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437844    5061 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:42:26.467477 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437847    5061 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:42:26.467477 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437849    5061 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:42:26.467477 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437852    5061 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:42:26.467477 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437855    5061 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:42:26.467477 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437859    5061 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:42:26.467477 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437862    5061 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:42:26.467477 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437866    5061 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:42:26.467477 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437870    5061 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:42:26.467477 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437873    5061 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:42:26.467477 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437876    5061 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:42:26.468313 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437878    5061 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:42:26.468313 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437881    5061 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:42:26.468313 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437884    5061 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:42:26.468313 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437886    5061 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:42:26.468313 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437889    5061 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:42:26.468313 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437892    5061 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:42:26.468313 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437894    5061 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:42:26.468313 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437898    5061 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:42:26.468313 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437902    5061 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:42:26.468313 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437905    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:42:26.468313 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437908    5061 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:42:26.468313 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437911    5061 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:42:26.468313 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437915    5061 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:42:26.468313 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437918    5061 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:42:26.468313 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437920    5061 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:42:26.468313 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437923    5061 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:42:26.468313 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437925    5061 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:42:26.468313 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437928    5061 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:42:26.468313 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437930    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:42:26.469190 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437938    5061 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:42:26.469190 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437941    5061 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:42:26.469190 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437944    5061 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:42:26.469190 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437946    5061 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:42:26.469190 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437949    5061 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:42:26.469190 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437951    5061 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:42:26.469190 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437954    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:42:26.469190 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437956    5061 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:42:26.469190 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437959    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:42:26.469190 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437961    5061 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:42:26.469190 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437965    5061 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:42:26.469190 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437968    5061 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:42:26.469190 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437971    5061 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:42:26.469190 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437973    5061 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:42:26.469190 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437976    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:42:26.469190 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437978    5061 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:42:26.469190 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437981    5061 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:42:26.469190 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437983    5061 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:42:26.469190 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437986    5061 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:42:26.469958 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437988    5061 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:42:26.469958 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437992    5061 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:42:26.469958 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437994    5061 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:42:26.469958 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437997    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:42:26.469958 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.437999    5061 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:42:26.469958 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.438002    5061 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:42:26.469958 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.438004    5061 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:42:26.469958 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:26.438007    5061 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:42:26.469958 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.438063    5061 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 21:42:26.469958 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.439105    5061 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 30 21:42:26.469958 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.439153    5061 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 30 21:42:26.469958 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.439360    5061 server.go:1014] "Starting client certificate rotation"
Oct 30 21:42:26.469958 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.439619    5061 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 21:42:26.469958 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.439995    5061 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 21:42:26.469958 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.439678    5061 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 21:36:45 +0000 UTC" deadline="2028-06-08 04:26:50.774508865 +0000 UTC"
Oct 30 21:42:26.469958 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.440204    5061 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14070h44m24.334308877s"
Oct 30 21:42:26.470621 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.441501    5061 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 21:42:26.470621 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.443168    5061 log.go:25] "Validated CRI v1 runtime API"
Oct 30 21:42:26.470621 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.445709    5061 log.go:25] "Validated CRI v1 image API"
Oct 30 21:42:26.470621 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.446389    5061 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 21:42:26.470621 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.448548    5061 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 a2fd7be1-fe1e-4e94-93c3-74428fedeaca:/dev/nvme0n1p3 fb547cbe-efef-43c5-9742-090921dd19a8:/dev/nvme0n1p4]
Oct 30 21:42:26.470723 ip-10-0-130-60 systemd[1]: Started Kubernetes Kubelet.
Oct 30 21:42:26.470959 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.448564    5061 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/087dbdeaadb724f2107d6c79a0edb74fa79f9fb572925a89d6c27a8e804aa75e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/087dbdeaadb724f2107d6c79a0edb74fa79f9fb572925a89d6c27a8e804aa75e/userdata/shm major:0 minor:296 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/23d53277e0647a0a05ba276cdb89af8108237e92f2dae2b39a5e3f45d1452f89/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/23d53277e0647a0a05ba276cdb89af8108237e92f2dae2b39a5e3f45d1452f89/userdata/shm major:0 minor:61 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3d5727e52b59c14557e36a0b0d4361bfd0b88a8203162fddca3943e12183fcbc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3d5727e52b59c14557e36a0b0d4361bfd0b88a8203162fddca3943e12183fcbc/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4276dc97fd6bbdde5d02f3244ef251d2b85ea8637cc7e47e2d6a5d871405e8cc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4276dc97fd6bbdde5d02f3244ef251d2b85ea8637cc7e47e2d6a5d871405e8cc/userdata/shm major:0 minor:62 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4b41219f99fde72e7b604b8387125ad69cba783b893bfd96f2e27d0ff43c8716/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4b41219f99fde72e7b604b8387125ad69cba783b893bfd96f2e27d0ff43c8716/userdata/shm major:0 minor:330 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/627a6b6e27f45f5178d5ceb3ee924b28635f2e39baa1d10d1290585ff7940e73/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/627a6b6e27f45f5178d5ceb3ee924b28635f2e39baa1d10d1290585ff7940e73/userdata/shm major:0 minor:232 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/65c7594af33d218097c8d7c8fa42fea0d244f99a0e390588efdeab8bdbf9ff99/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/65c7594af33d218097c8d7c8fa42fea0d244f99a0e390588efdeab8bdbf9ff99/userdata/shm major:0 minor:293 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6fc99d3eb77d11207c5f020fdbd83137a53549d5a511203325cc56a00f45ea1c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6fc99d3eb77d11207c5f020fdbd83137a53549d5a511203325cc56a00f45ea1c/userdata/shm major:0 minor:234 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7429a4fe42cd3da457a0e804b878c66475785612023343f677f33b799c6bf1ac/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7429a4fe42cd3da457a0e804b878c66475785612023343f677f33b799c6bf1ac/userdata/shm major:0 minor:252 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/75e0ccc2236e8a6d94ca8547c5e6e12b14a26ebd6ef7344628e37e23a7e8601f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/75e0ccc2236e8a6d94ca8547c5e6e12b14a26ebd6ef7344628e37e23a7e8601f/userdata/shm major:0 minor:221 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/76ef14d66ab9af8383c8ea83abcbfb7325220d8b7e326ef09c6a16fab7db51b2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/76ef14d66ab9af8383c8ea83abcbfb7325220d8b7e326ef09c6a16fab7db51b2/userdata/shm major:0 minor:60 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/82a9465e06fce6f349977da268c5b1d09b7af9519dedc01b9ebda34e5f224703/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/82a9465e06fce6f349977da268c5b1d09b7af9519dedc01b9ebda34e5f224703/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8da43005c1309e0d14d60d540ea37689ef09c76768e5ae580bcfed68f38e70e7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8da43005c1309e0d14d60d540ea37689ef09c76768e5ae580bcfed68f38e70e7/userdata/shm major:0 minor:362 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a0b977d201619f90b14be079450e973daf8fbde1644152a642ffd5b9ad30c498/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a0b977d201619f90b14be079450e973daf8fbde1644152a642ffd5b9ad30c498/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a55e6c85fda472364c013d6264207e438910511591249633978f190007eae893/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a55e6c85fda472364c013d6264207e438910511591249633978f190007eae893/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a8dee9b06f8070907d581c085c189097c986b3347dddd64dfe1637cd79b4352c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a8dee9b06f8070907d581c085c189097c986b3347dddd64dfe1637cd79b4352c/userdata/shm major:0 minor:63 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b1588b1f5ed94ade8da31ac44f2630d4b83ff1eb7c553d3e27b4619894d66f45/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b1588b1f5ed94ade8da31ac44f2630d4b83ff1eb7c553d3e27b4619894d66f45/userdata/shm major:0 minor:236 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/be70abae34436026075497a968e2b20eea71e3e611aa0c26f209a31ef280dd9a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/be70abae34436026075497a968e2b20eea71e3e611aa0c26f209a31ef280dd9a/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c20b0427474a2038c6010808a9c237275e18c16b1f9fa4fa8873f8277f6abe2b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c20b0427474a2038c6010808a9c237275e18c16b1f9fa4fa8873f8277f6abe2b/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c25682648a4da3674e83bed18c8a7c4f12a24442876464de11bece9d61180c06/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c25682648a4da3674e83bed18c8a7c4f12a24442876464de11bece9d61180c06/userdata/shm major:0 minor:344 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c5dd861e453dd124798e665be270326747c601f40e1402e9d28e79daf84c5712/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c5dd861e453dd124798e665be270326747c601f40e1402e9d28e79daf84c5712/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cd13c3e7b0eb8ad54aed3de1a1bb8c6b1edc99e29a2368dcc6d659c394591cfc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cd13c3e7b0eb8ad54aed3de1a1bb8c6b1edc99e29a2368dcc6d659c394591cfc/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d6179def9b341a17c6786ff22e0769ea943a3b99f9e3f8d3aaf621d338cb0ae6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d6179def9b341a17c6786ff22e0769ea943a3b99f9e3f8d3aaf621d338cb0ae6/userdata/shm major:0 minor:213 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e369a0f022d80f14a70bcfff6d7599bae3b42ee10574fb6f745e5f9bcc7fa2ad/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e369a0f022d80f14a70bcfff6d7599bae3b42ee10574fb6f745e5f9bcc7fa2ad/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e94508390d125c0485323172c4f0b87688846973bfa23f1a4a6baa825ea13544/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e94508390d125c0485323172c4f0b87688846973bfa23f1a4a6baa825ea13544/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fd3c3fb68c3b02cc83aea080da102d52022e8b0f93fdcaa3578ed0d47045be84/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fd3c3fb68c3b02cc83aea080da102d52022e8b0f93fdcaa3578ed0d47045be84/userdata/shm major:0 minor:336 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0126a0a0-a1c8-4db3-a6af-629360e247f4/volumes/kubernetes.io~projected/kube-api-access-srwt7:{mountpoint:/var/lib/kubelet/pods/0126a0a0-a1c8-4db3-a6af-629360e247f4/volumes/kubernetes.io~projected/kube-api-access-srwt7 major:0 minor:231 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0126a0a0-a1c8-4db3-a6af-629360e247f4/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/0126a0a0-a1c8-4db3-a6af-629360e247f4/volumes/kubernetes.io~secret/serving-cert major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/033356b5-3c05-4f49-ad6a-7c369259970d/volumes/kubernetes.io~projected/kube-api-access-d9grk:{mountpoint:/var/lib/kubelet/pods/033356b5-3c05-4f49-ad6a-7c369259970d/volumes/kubernetes.io~projected/kube-api-access-d9grk major:0 minor:327 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0929b9f6-12ec-4f4d-8ba3-5cef9defba15/volumes/kubernetes.io~projected/kube-api-access-swjcw:{mountpoint:/var/lib/kubelet/pods/0929b9f6-12ec-4f4d-8ba3-5cef9defba15/volumes/kubernetes.io~projected/kube-api-access-swjcw major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0bb51175-fa5e-4472-8d83-106bca721585/volumes/kubernetes.io~projected/kube-api-access-pdbmm:{mountpoint:/var/lib/kubelet/pods/0bb51175-fa5e-4472-8d83-106bca721585/volumes/kubernetes.io~projected/kube-api-access-pdbmm major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/101c697b-d7b8-4cfe-9d4c-818b5c04108f/volumes/kubernetes.io~projected/kube-api-access-xm5l5:{mountpoint:/var/lib/kubelet/pods/101c697b-d7b8-4cfe-9d4c-818b5c04108f/volumes/kubernetes.io~projected/kube-api-access-xm5l5 major:0 minor:343 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/102b8049-7349-4e2b-974a-44c901eebcf3/volumes/kubernetes.io~projected/kube-api-access-5fcz7:{mountpoint:/var/lib/kubelet/pods/102b8049-7349-4e2b-974a-44c901eebcf3/volumes/kubernetes.io~projected/kube-api-access-5fcz7 major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1f3b6a23-d7ea-4ce4-b1e8-68133c4b4ebc/volumes/kubernetes.io~projected/kube-api-access-7gq9m:{mountpoint:/var/lib/kubelet/pods/1f3b6a23-d7ea-4ce4-b1e8-68133c4b4ebc/volumes/kubernetes.io~projected/kube-api-access-7gq9m major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/20f55ce3-5b04-4916-b529-0dd596aba68f/volumes/kubernetes.io~projected/kube-api-access-kr7ww:{mountpoint:/var/lib/kubelet/pods/20f55ce3-5b04-4916-b529-0dd596aba68f/volumes/kubernetes.io~projected/kube-api-access-kr7ww major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2205e8e6-f038-47fa-b393-0abe4d622e03/volumes/kubernetes.io~projected/kube-api-access-rbwzr:{mountpoint:/var/lib/kubelet/pods/2205e8e6-f038-47fa-b393-0abe4d622e03/volumes/kubernetes.io~projected/kube-api-access-rbwzr major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/238ff7d8-7da8-4cd8-9d45-deedf615211e/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/238ff7d8-7da8-4cd8-9d45-deedf615211e/volumes/kubernetes.io~secret/agent-certs major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/28210ad9-18f0-4ecf-9bfc-b256440f0ce8/volumes/kubernetes.io~projected/kube-api-access-l75p7:{mountpoint:/var/lib/kubelet/pods/28210ad9-18f0-4ecf-9bfc-b256440f0ce8/volumes/kubernetes.io~projected/kube-api-access-l75p7 major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/28210ad9-18f0-4ecf-9bfc-b256440f0ce8/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/28210ad9-18f0-4ecf-9bfc-b256440f0ce8/volumes/kubernetes.io~secret/cert major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3784e680-5efe-4ffb-ad71-956d05b57dce/volumes/kubernetes.io~projected/kube-api-access-579v9:{mountpoint:/var/lib/kubelet/pods/3784e680-5efe-4ffb-ad71-956d05b57dce/volumes/kubernetes.io~projected/kube-api-access-579v9 major:0 minor:295 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3f97ee34-5dfc-454e-ba5c-afd939f3bd25/volumes/kubernetes.io~projected/kube-api-access-cdtc5:{mountpoint:/var/lib/kubelet/pods/3f97ee34-5dfc-454e-ba5c-afd939f3bd25/volumes/kubernetes.io~projected/kube-api-access-cdtc5 major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4427f59d-8003-440f-9646-a46b176cc4a7/volumes/kubernetes.io~projected/kube-api-access-nc76p:{mountpoint:/var/lib/kubelet/pods/4427f59d-8003-440f-9646-a46b176cc4a7/volumes/kubernetes.io~projected/kube-api-access-nc76p major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/538d5103-1c92-48f1-acc7-f90956848727/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/538d5103-1c92-48f1-acc7-f90956848727/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/538d5103-1c92-48f1-acc7-f90956848727/volumes/kubernetes.io~projected/kube-api-access-hrpz5:{mountpoint:/var/lib/kubelet/pods/538d5103-1c92-48f1-acc7-f90956848727/volumes/kubernetes.io~projected/kube-api-access-hrpz5 major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/538d5103-1c92-48f1-acc7-f90956848727/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/538d5103-1c92-48f1-acc7-f90956848727/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5e74071e-0462-42b5-a05c-d90bb638478a/volumes/kubernetes.io~projected/kube-api-access-vqmr4:{mountpoint:/var/lib/kubelet/pods/5e74071e-0462-42b5-a05c-d90bb638478a/volumes/kubernetes.io~projected/kube-api-access-vqmr4 major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5e74071e-0462-42b5-a05c-d90bb638478a/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/5e74071e-0462-42b5-a05c-d90bb638478a/volumes/kubernetes.io~secret/serving-cert major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/613e7c8a-e2c7-424a-93e0-7fa86d18a8a9/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/613e7c8a-e2c7-424a-93e0-7fa86d18a8a9/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/613e7c8a-e2c7-424a-93e0-7fa86d18a8a9/volumes/kubernetes.io~projected/kube-api-access-hw226:{mountpoint:/var/lib/kubelet/pods/613e7c8a-e2c7-424a-93e0-7fa86d18a8a9/volumes/kubernetes.io~projected/kube-api-access-hw226 major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/613e7c8a-e2c7-424a-93e0-7fa86d18a8a9/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/613e7c8a-e2c7-424a-93e0-7fa86d18a8a9/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/613e7c8a-e2c7-424a-93e0-7fa86d18a8a9/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/613e7c8a-e2c7-424a-93e0-7fa86d18a8a9/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/71ddf455-cbf7-49a6-9773-30f7e4997a39/volumes/kubernetes.io~projected/kube-api-access-7hkfh:{mountpoint:/var/lib/kubelet/pods/71ddf455-cbf7-49a6-9773-30f7e4997a39/volumes/kubernetes.io~projected/kube-api-access-7hkfh major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/74ff988a-c05e-4a3e-adff-5945b79cf248/volumes/kubernetes.io~projected/kube-api-access-67qtb:{mountpoint:/var/lib/kubelet/pods/74ff988a-c05e-4a3e-adff-5945b79cf248/volumes/kubernetes.io~projected/kube-api-access-67qtb major:0 minor:338 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/798f248f-d429-4f56-8752-e157dd94206b/volumes/kubernetes.io~projected/kube-api-access-8w7g2:{mountpoint:/var/lib/kubelet/pods/798f248f-d429-4f56-8752-e157dd94206b/volumes/kubernetes.io~projected/kube-api-access-8w7g2 major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/798f248f-d429-4f56-8752-e157dd94206b/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/798f248f-d429-4f56-8752-e157dd94206b/volumes/kubernetes.io~secret/serving-cert major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/84d9b682-3241-40af-87d5-f87d54ce3697/volumes/kubernetes.io~projected/kube-api-access-mms6q:{mountpoint:/var/lib/kubelet/pods/84d9b682-3241-40af-87d5-f87d54ce3697/volumes/kubernetes.io~projected/kube-api-access-mms6q major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/879022d4-06b9-44cc-a2a2-dc307b27345e/volumes/kubernetes.io~projected/kube-api-access-4658n:{mountpoint:/var/lib/kubelet/pods/879022d4-06b9-44cc-a2a2-dc307b27345e/volumes/kubernetes.io~projected/kube-api-access-4658n major:0 minor:322 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d/volumes/kubernetes.io~projected/kube-api-access-n4tb7:{mountpoint:/var/lib/kubelet/pods/9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d/volumes/kubernetes.io~projected/kube-api-access-n4tb7 major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d/volumes/kubernetes.io~secret/default-certificate major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d/volumes/kubernetes.io~secret/stats-auth major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a2e3b25e-b6dc-46ef-86e8-4e70f9dad200/volumes/kubernetes.io~projected/kube-api-access-p8b7v:{mountpoint:/var/lib/kubelet/pods/a2e3b25e-b6dc-46ef-86e8-4e70f9dad200/volumes/kubernetes.io~projected/kube-api-access-p8b7v major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a32aacdc-53be-41cf-839e-435d43361890/volumes/kubernetes.io~projected/kube-api-access-424gm:{mountpoint:/var/lib/kubelet/pods/a32aacdc-53be-41cf-839e-435d43361890/volumes/kubernetes.io~projected/kube-api-access-424gm major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b227ad6a-9e59-415e-b7e6-f7c25e988abb/volumes/kubernetes.io~projected/kube-api-access-qpbtv:{mountpoint:/var/lib/kubelet/pods/b227ad6a-9e59-415e-b7e6-f7c25e988abb/volumes/kubernetes.io~projected/kube-api-access-qpbtv major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c2386477-903e-4c6c-b807-3f473a58a6b6/volumes/kubernetes.io~projected/kube-api-access-zv2qw:{mountpoint:/var/lib/kubelet/pods/c2386477-903e-4c6c-b807-3f473a58a6b6/volumes/kubernetes.io~projected/kube-api-access-zv2qw major:0 minor:361 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c2386477-903e-4c6c-b807-3f473a58a6b6/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/c2386477-903e-4c6c-b807-3f473a58a6b6/volumes/kubernetes.io~secret/signing-key major:0 minor:356 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c69312da-8be3-4931-8a81-a8ff3f86d5e6/volumes/kubernetes.io~projected/kube-api-access-qd5b8:{mountpoint:/var/lib/kubelet/pods/c69312da-8be3-4931-8a81-a8ff3f86d5e6/volumes/kubernetes.io~projected/kube-api-access-qd5b8 major:0 minor:228 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c69312da-8be3-4931-8a81-a8ff3f86d5e6/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/c69312da-8be3-4931-8a81-a8ff3f86d5e6/volumes/kubernetes.io~secret/serving-cert major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c798fd57-e0be-4b61-860b-0b833aca7784/volumes/kubernetes.io~projected/kube-api-access-xk5zb:{mountpoint:/var/lib/kubelet/pods/c798fd57-e0be-4b61-860b-0b833aca7784/volumes/kubernetes.io~projected/kube-api-access-xk5zb major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cad67bc5-c8d2-497c-ba1b-cdbae5822cfa/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/cad67bc5-c8d2-497c-ba1b-cdbae5822cfa/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cad67bc5-c8d2-497c-ba1b-cdbae5822cfa/volumes/kubernetes.io~projected/kube-api-access-m9dpv:{mountpoint:/var/lib/kubelet/pods/cad67bc5-c8d2-497c-ba1b-cdbae5822cfa/volumes/kubernetes.io~projected/kube-api-access-m9dpv major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cad67bc5-c8d2-497c-ba1b-cdbae5822cfa/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/cad67bc5-c8d2-497c-ba1b-cdbae5822cfa/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cad67bc5-c8d2-497c-ba1b-cdbae5822cfa/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/cad67bc5-c8d2-497c-ba1b-cdbae5822cfa/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d700d1eb-fbf4-45df-9b57-eca08d87ebe2/volumes/kubernetes.io~projected/kube-api-access-nhbrj:{mountpoint:/var/lib/kubelet/pods/d700d1eb-fbf4-45df-9b57-eca08d87ebe2/volumes/kubernetes.io~projected/kube-api-access-nhbrj major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ed08bc51-70e6-4658-adb3-ed36f5176552/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/ed08bc51-70e6-4658-adb3-ed36f5176552/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ed08bc51-70e6-4658-adb3-ed36f5176552/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/ed08bc51-70e6-4658-adb3-ed36f5176552/volumes/kubernetes.io~empty-dir/tmp major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ed08bc51-70e6-4658-adb3-ed36f5176552/volumes/kubernetes.io~projected/kube-api-access-pd6bv:{mountpoint:/var/lib/kubelet/pods/ed08bc51-70e6-4658-adb3-ed36f5176552/volumes/kubernetes.io~projected/kube-api-access-pd6bv major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fe41c70b-7e2c-432f-8389-0282d6fdb332/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/fe41c70b-7e2c-432f-8389-0282d6fdb332/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:288 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/6e050dc65df7774fc1933df24ff25abdcc8e3727099d3e3a04ca7d635b20fb5c/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/fae30d5e094612debe3b2efd1ca4d61e8d9168d182739d9d8a88978d99dad45f/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/1f39a939ee073eaac09dcbc51b19999b75e8f8af6785d3e05a80d460aec5b79a/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/9f8704300d4cf99b1e653e7ee523b6e282a1c7d79f4fe9814f7152e001429cb5/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-111:{mountpoint:/var/lib/containers/storage/overlay/1a7b434e29583aeb415aec661862308757a0fdf8291b0cd910a0ea5979f848f0/merged major:0 minor:111 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/06f0cb83331b396698e99e9a17042dc4feb25a5758aa1944c13ea3635de64996/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/9ac3d9dd7ed5f3732580793f38f9a144d8b8e6808ef1fcd7c18ed1318e714601/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/1a037545e17a689e640d568c9c1c61d73b3c106164750d63a00a71fad1490a2b/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/b7a058accb525a32ffe36ea7775b85bbdc0cd3bfaad591acea78eb305dc74cdc/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/a08dfbfc0f60a854c12432bcaea91b623516eff969547a53205d7e225cdad8dc/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/3bf8f6c705520bd4992ef35efad5b77166d70735d0a396ac05e9fc193aaff038/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/d5858c01a017c8e839102db5a9970aae8286b7caf9355322049514e0c22755e3/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/1912d2976aca7c7c1233a58b612c64ef529db2e90674cbfea8d9ac90b4af5a2a/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/931319e8f08fcabe7bb6d023d5f397db1d628e609d6e14cc883722eef5c904b1/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/cd374ed2b85ff69c770ee08ece963a3b6aa4cb05a2e4b83ae17d6ec9a84b5af3/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/d5fb28316b2ece56255958d3e51bc174b92d0a14a255e16bb2ac4f2088196522/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/03e4c2f88499c09429ebe2c2dea2657d67883c14f69d18aa30dbbce88770ac99/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/9124bd75578df6950f897c8629014121952680fb213e4cc4b3233e325c5d8198/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/f4387434f95f5b0b081ac87f692169cb79fa7687f5bdfc8a7974cd602045f4de/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/9e9d5a582b6e2cc704a8971d822bd3351b80a0c9f02ba31d1ef080b79ab3833a/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-191:{mountpoint:/var/lib/containers/storage/overlay/931d4c6b24964e9b28ddf14919dd429d97f1c8522610ae46369887195ff64ccc/merged major:0 minor:191 fsType:overlay blockSize:0} overlay_0-246:{mountpoint:/var/lib/containers/storage/overlay/6bf7e2748d0c50ed4ffbbf27b0c2509911b0c12c58ac0cb6b83293329e57325c/merged major:0 minor:246 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/4a70129b7a58caf812f8c1c1ccbec6e59b41948486a9e3abe41815b6fbd567e8/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/728fc77ef0d9f92c840574b828453a31dccec3d8a3e54cd11023989e6b55450c/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/f84510d3d3b6cf091443784b8d94326500043d3407bfab364c445d3a89411b20/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/444d9141c7eeed75f7164b1dc712fcd54c41ab1c2a7486ceb628b57439ec59de/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/d8f2eec3346d431b9a25df26cd05546540c92bc64769d5d2c3d180f1baca3ac6/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/adf4aaf6cb7dc1cd465a1f6251ae0bdd302a142aefeebba298db5642c1bc95d3/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/bd50655d5764783f0bd06fff08c4e966c652c3b9be3f4305a5ec52c38969457d/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-268:{mountpoint:/var/lib/containers/storage/overlay/91db42615f9aa81a19c545da0733c2af557ec054b67165ad7f01f9105a599286/merged major:0 minor:268 fsType:overlay blockSize:0} overlay_0-270:{mountpoint:/var/lib/containers/storage/overlay/b45e13b0fd1f9465ca86c239f5e72f0e2689aa0edc00e15211a57b1a684ad7b5/merged major:0 minor:270 fsType:overlay blockSize:0} overlay_0-272:{mountpoint:/var/lib/containers/storage/overlay/f2ec330bbecf1064aa41ee75eb7af01e0e4e04a8d9cb967f0eece96f612a1812/merged major:0 minor:272 fsType:overlay blockSize:0} overlay_0-274:{mountpoint:/var/lib/containers/storage/overlay/7298f20e10edaf7a01d8cdeb7c624b7ab0effe6971ffd57f39fc4f1a180941b3/merged major:0 minor:274 fsType:overlay blockSize:0} overlay_0-284:{mountpoint:/var/lib/containers/storage/overlay/7528e60d141eee5886f9fad6214433e10022dbbd60041574d7db84036c08ed38/merged major:0 minor:284 fsType:overlay blockSize:0} overlay_0-286:{mountpoint:/var/lib/containers/storage/overlay/9117af8252b56201ed230f3b9020820a243169ec382fc261ba4dad8d8550bb26/merged major:0 minor:286 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/9b321664c10a751b0e380a945236533430c34be1103f3b49c8bd00a2dbe66d5a/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/ed2d91f466f9548cbf1b021e60b64d51587728dfe83399f69ee6fae19bf11fe2/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/fc8ae448c77f03194cf36edbd54c17b45758ac1205a18ec1f6b45f893fec6d5a/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/9a66cb6e6b5b0b9b11ec5615277261ae17d8a456076c5ebe31083ad113b97299/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-310:{mountpoint:/var/lib/containers/storage/overlay/92896ac0fabdcfb228f35941d356d7a4c5f596f0b28139ff5843d395998c637c/merged major:0 minor:310 fsType:overlay blockSize:0} overlay_0-312:{mountpoint:/var/lib/containers/storage/overlay/ca5784c1b15d87b786c82702635eeb377c334b9fc3fea0ae760f3fb45aa9a0b7/merged major:0 minor:312 fsType:overlay blockSize:0} overlay_0-328:{mountpoint:/var/lib/containers/storage/overlay/17d39bd2ef51ca756413dfd073097146d43b459ec61879bd57b8cb870f8b08bb/merged major:0 minor:328 fsType:overlay blockSize:0} overlay_0-339:{mountpoint:/var/lib/containers/storage/overlay/975d1e9f7059aef3bb10cbed48ec3e7989bb9e254eab8e0c1be10de94ce20de0/merged major:0 minor:339 fsType:overlay blockSize:0} overlay_0-341:{mountpoint:/var/lib/containers/storage/overlay/bc38ef94ba1c85e13a2bb9a9bef604aef210ec9c2845864c773f7fc0e7f09857/merged major:0 minor:341 fsType:overlay blockSize:0} overlay_0-346:{mountpoint:/var/lib/containers/storage/overlay/22b4170ea7a94636835236f73f18611abc2030c8c33c30c6fd76f276ca41230f/merged major:0 minor:346 fsType:overlay blockSize:0} overlay_0-348:{mountpoint:/var/lib/containers/storage/overlay/1870b6480d4aa9008238961f78b9d83a81fb05da3543d554a51e210ab5287d1f/merged major:0 minor:348 fsType:overlay blockSize:0} overlay_0-354:{mountpoint:/var/lib/containers/storage/overlay/d904018fe814268e87f15789a919832f2964c9e6a28cf87fdbc49ca792fb8fc0/merged major:0 minor:354 fsType:overlay blockSize:0} overlay_0-364:{mountpoint:/var/lib/containers/storage/overlay/9fcd45054cb18cf001e411a1fdf7cbb5f831bd0bca5b4fac3d57fdbea5b52d64/merged major:0 minor:364 fsType:overlay blockSize:0} overlay_0-366:{mountpoint:/var/lib/containers/storage/overlay/b7f6d62ea8961f0f94ddc295f297f37c3f607c776f90436845f3b6bc3432c340/merged major:0 minor:366 fsType:overlay blockSize:0} overlay_0-368:{mountpoint:/var/lib/containers/storage/overlay/0684b9f7023351d545ed9dffe03cf0eef4a4910b7027e8c1313301b2e4beaba4/merged major:0 minor:368 fsType:overlay blockSize:0} overlay_0-373:{mountpoint:/var/lib/containers/storage/overlay/1a1f94ce20953a7f40139e6b421d1b376c6c931fb4a4e20dae9c54ae342d40ff/merged major:0 minor:373 fsType:overlay blockSize:0} overlay_0-376:{mountpoint:/var/lib/containers/storage/overlay/794275cd6eff2903af925fc5c4c93f17f008d4ba881f03e317bda0d8d60636de/merged major:0 minor:376 fsType:overlay blockSize:0} overlay_0-68:{mountpoint:/var/lib/containers/storage/overlay/92d9027678b2eaaf785ce18de0f520f504a842dc0c906848bca9cdaef1b061cd/merged major:0 minor:68 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/f22130ce751955a5cbae91d079d988db54feb5ed343db58c85ac236db2757eca/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/617a4e995f2825f0d525c0a9919b1d3b4be8b22a991d247c0244b1ab1bbe3ca8/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/a25b8807ef8c183733b3ddcceecf68e4b9299d49311072b3d8875600b8201c86/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/4550bafc951684645e018d3614a9c57f291f1d7990a923cfb47f7da60dd538fd/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/a99c44f90e69e5a57e096d8713b8df72c3b2f747dc590bc81cf95c1481c8d5ef/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/e2fec555fb0e24ec38ad13316f6cb2fefbb834b578c6ddc1301860e29f1fddb9/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/6d428869419663acf63642d4b5c15e53d55b072c87d1c59b42cd02b86510ed52/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/2e35e1a0a66f47a7f6d5e5592d1e7922768709e666c5ce17c9316a90964ab611/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/7bc7cbc837bbe01bef49b20682a8814e5ccadb5a600dbe6caf2d5041412554a4/merged major:0 minor:96 fsType:overlay blockSize:0}]
Oct 30 21:42:26.471327 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.459417    5061 manager.go:217] Machine: {Timestamp:2026-10-30 21:42:26.458945368 +0000 UTC m=+0.071715117 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099937 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2d41036d8f146a8fdfb95efa4ac9ad SystemUUID:ec2d4103-6d8f-146a-8fdf-b95efa4ac9ad BootID:db6d970d-f626-47d3-94e1-8b765db49ded Filesystems:[{Device:/var/lib/kubelet/pods/9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d/volumes/kubernetes.io~projected/kube-api-access-n4tb7 DeviceMajor:0 DeviceMinor:206 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/033356b5-3c05-4f49-ad6a-7c369259970d/volumes/kubernetes.io~projected/kube-api-access-d9grk DeviceMajor:0 DeviceMinor:327 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c25682648a4da3674e83bed18c8a7c4f12a24442876464de11bece9d61180c06/userdata/shm DeviceMajor:0 DeviceMinor:344 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/a2e3b25e-b6dc-46ef-86e8-4e70f9dad200/volumes/kubernetes.io~projected/kube-api-access-p8b7v DeviceMajor:0 DeviceMinor:59 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/cad67bc5-c8d2-497c-ba1b-cdbae5822cfa/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:225 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-68 DeviceMajor:0 DeviceMinor:68 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ed08bc51-70e6-4658-adb3-ed36f5176552/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:47 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/538d5103-1c92-48f1-acc7-f90956848727/volumes/kubernetes.io~projected/kube-api-access-hrpz5 DeviceMajor:0 DeviceMinor:57 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/613e7c8a-e2c7-424a-93e0-7fa86d18a8a9/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:45 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/cad67bc5-c8d2-497c-ba1b-cdbae5822cfa/volumes/kubernetes.io~projected/kube-api-access-m9dpv DeviceMajor:0 DeviceMinor:223 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-274 DeviceMajor:0 DeviceMinor:274 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-286 DeviceMajor:0 DeviceMinor:286 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/76ef14d66ab9af8383c8ea83abcbfb7325220d8b7e326ef09c6a16fab7db51b2/userdata/shm DeviceMajor:0 DeviceMinor:60 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/28210ad9-18f0-4ecf-9bfc-b256440f0ce8/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:218 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3784e680-5efe-4ffb-ad71-956d05b57dce/volumes/kubernetes.io~projected/kube-api-access-579v9 DeviceMajor:0 DeviceMinor:295 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/74ff988a-c05e-4a3e-adff-5945b79cf248/volumes/kubernetes.io~projected/kube-api-access-67qtb DeviceMajor:0 DeviceMinor:338 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-366 DeviceMajor:0 DeviceMinor:366 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5e74071e-0462-42b5-a05c-d90bb638478a/volumes/kubernetes.io~projected/kube-api-access-vqmr4 DeviceMajor:0 DeviceMinor:226 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-268 DeviceMajor:0 DeviceMinor:268 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-373 DeviceMajor:0 DeviceMinor:373 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/538d5103-1c92-48f1-acc7-f90956848727/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/798f248f-d429-4f56-8752-e157dd94206b/volumes/kubernetes.io~projected/kube-api-access-8w7g2 DeviceMajor:0 DeviceMinor:205 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c2386477-903e-4c6c-b807-3f473a58a6b6/volumes/kubernetes.io~projected/kube-api-access-zv2qw DeviceMajor:0 DeviceMinor:361 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1f3b6a23-d7ea-4ce4-b1e8-68133c4b4ebc/volumes/kubernetes.io~projected/kube-api-access-7gq9m DeviceMajor:0 DeviceMinor:58 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:203 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-376 DeviceMajor:0 DeviceMinor:376 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4427f59d-8003-440f-9646-a46b176cc4a7/volumes/kubernetes.io~projected/kube-api-access-nc76p DeviceMajor:0 DeviceMinor:51 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/28210ad9-18f0-4ecf-9bfc-b256440f0ce8/volumes/kubernetes.io~projected/kube-api-access-l75p7 DeviceMajor:0 DeviceMinor:229 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7429a4fe42cd3da457a0e804b878c66475785612023343f677f33b799c6bf1ac/userdata/shm DeviceMajor:0 DeviceMinor:252 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-312 DeviceMajor:0 DeviceMinor:312 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/101c697b-d7b8-4cfe-9d4c-818b5c04108f/volumes/kubernetes.io~projected/kube-api-access-xm5l5 DeviceMajor:0 DeviceMinor:343 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/ed08bc51-70e6-4658-adb3-ed36f5176552/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:49 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/538d5103-1c92-48f1-acc7-f90956848727/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:50 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/23d53277e0647a0a05ba276cdb89af8108237e92f2dae2b39a5e3f45d1452f89/userdata/shm DeviceMajor:0 DeviceMinor:61 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cd13c3e7b0eb8ad54aed3de1a1bb8c6b1edc99e29a2368dcc6d659c394591cfc/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c69312da-8be3-4931-8a81-a8ff3f86d5e6/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:216 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/cad67bc5-c8d2-497c-ba1b-cdbae5822cfa/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:217 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/20f55ce3-5b04-4916-b529-0dd596aba68f/volumes/kubernetes.io~projected/kube-api-access-kr7ww DeviceMajor:0 DeviceMinor:224 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c5dd861e453dd124798e665be270326747c601f40e1402e9d28e79daf84c5712/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a55e6c85fda472364c013d6264207e438910511591249633978f190007eae893/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-348 DeviceMajor:0 DeviceMinor:348 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3d5727e52b59c14557e36a0b0d4361bfd0b88a8203162fddca3943e12183fcbc/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/613e7c8a-e2c7-424a-93e0-7fa86d18a8a9/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:204 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/613e7c8a-e2c7-424a-93e0-7fa86d18a8a9/volumes/kubernetes.io~projected/kube-api-access-hw226 DeviceMajor:0 DeviceMinor:207 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/82a9465e06fce6f349977da268c5b1d09b7af9519dedc01b9ebda34e5f224703/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2205e8e6-f038-47fa-b393-0abe4d622e03/volumes/kubernetes.io~projected/kube-api-access-rbwzr DeviceMajor:0 DeviceMinor:54 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/798f248f-d429-4f56-8752-e157dd94206b/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:202 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b1588b1f5ed94ade8da31ac44f2630d4b83ff1eb7c553d3e27b4619894d66f45/userdata/shm DeviceMajor:0 DeviceMinor:236 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fd3c3fb68c3b02cc83aea080da102d52022e8b0f93fdcaa3578ed0d47045be84/userdata/shm DeviceMajor:0 DeviceMinor:336 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c2386477-903e-4c6c-b807-3f473a58a6b6/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:356 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/71ddf455-cbf7-49a6-9773-30f7e4997a39/volumes/kubernetes.io~projected/kube-api-access-7hkfh DeviceMajor:0 DeviceMinor:56 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/613e7c8a-e2c7-424a-93e0-7fa86d18a8a9/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:197 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0126a0a0-a1c8-4db3-a6af-629360e247f4/volumes/kubernetes.io~projected/kube-api-access-srwt7 DeviceMajor:0 DeviceMinor:231 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-364 DeviceMajor:0 DeviceMinor:364 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/75e0ccc2236e8a6d94ca8547c5e6e12b14a26ebd6ef7344628e37e23a7e8601f/userdata/shm DeviceMajor:0 DeviceMinor:221 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a0b977d201619f90b14be079450e973daf8fbde1644152a642ffd5b9ad30c498/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0126a0a0-a1c8-4db3-a6af-629360e247f4/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:215 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c20b0427474a2038c6010808a9c237275e18c16b1f9fa4fa8873f8277f6abe2b/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/65c7594af33d218097c8d7c8fa42fea0d244f99a0e390588efdeab8bdbf9ff99/userdata/shm DeviceMajor:0 DeviceMinor:293 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-368 DeviceMajor:0 DeviceMinor:368 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-270 DeviceMajor:0 DeviceMinor:270 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e94508390d125c0485323172c4f0b87688846973bfa23f1a4a6baa825ea13544/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0bb51175-fa5e-4472-8d83-106bca721585/volumes/kubernetes.io~projected/kube-api-access-pdbmm DeviceMajor:0 DeviceMinor:227 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c798fd57-e0be-4b61-860b-0b833aca7784/volumes/kubernetes.io~projected/kube-api-access-xk5zb DeviceMajor:0 DeviceMinor:211 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d700d1eb-fbf4-45df-9b57-eca08d87ebe2/volumes/kubernetes.io~projected/kube-api-access-nhbrj DeviceMajor:0 DeviceMinor:212 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/fe41c70b-7e2c-432f-8389-0282d6fdb332/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:288 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-341 DeviceMajor:0 DeviceMinor:341 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8da43005c1309e0d14d60d540ea37689ef09c76768e5ae580bcfed68f38e70e7/userdata/shm DeviceMajor:0 DeviceMinor:362 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-111 DeviceMajor:0 DeviceMinor:111 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-191 DeviceMajor:0 DeviceMinor:191 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/102b8049-7349-4e2b-974a-44c901eebcf3/volumes/kubernetes.io~projected/kube-api-access-5fcz7 DeviceMajor:0 DeviceMinor:208 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-328 DeviceMajor:0 DeviceMinor:328 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4b41219f99fde72e7b604b8387125ad69cba783b893bfd96f2e27d0ff43c8716/userdata/shm DeviceMajor:0 DeviceMinor:330 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4276dc97fd6bbdde5d02f3244ef251d2b85ea8637cc7e47e2d6a5d871405e8cc/userdata/shm DeviceMajor:0 DeviceMinor:62 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-346 DeviceMajor:0 DeviceMinor:346 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0929b9f6-12ec-4f4d-8ba3-5cef9defba15/volumes/kubernetes.io~projected/kube-api-access-swjcw DeviceMajor:0 DeviceMinor:55 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-310 DeviceMajor:0 DeviceMinor:310 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b227ad6a-9e59-415e-b7e6-f7c25e988abb/volumes/kubernetes.io~projected/kube-api-access-qpbtv DeviceMajor:0 DeviceMinor:53 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/879022d4-06b9-44cc-a2a2-dc307b27345e/volumes/kubernetes.io~projected/kube-api-access-4658n DeviceMajor:0 DeviceMinor:322 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cad67bc5-c8d2-497c-ba1b-cdbae5822cfa/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:220 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/84d9b682-3241-40af-87d5-f87d54ce3697/volumes/kubernetes.io~projected/kube-api-access-mms6q DeviceMajor:0 DeviceMinor:230 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/627a6b6e27f45f5178d5ceb3ee924b28635f2e39baa1d10d1290585ff7940e73/userdata/shm DeviceMajor:0 DeviceMinor:232 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/087dbdeaadb724f2107d6c79a0edb74fa79f9fb572925a89d6c27a8e804aa75e/userdata/shm DeviceMajor:0 DeviceMinor:296 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-246 DeviceMajor:0 DeviceMinor:246 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/be70abae34436026075497a968e2b20eea71e3e611aa0c26f209a31ef280dd9a/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6fc99d3eb77d11207c5f020fdbd83137a53549d5a511203325cc56a00f45ea1c/userdata/shm DeviceMajor:0 DeviceMinor:234 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d6179def9b341a17c6786ff22e0769ea943a3b99f9e3f8d3aaf621d338cb0ae6/userdata/shm DeviceMajor:0 DeviceMinor:213 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c69312da-8be3-4931-8a81-a8ff3f86d5e6/volumes/kubernetes.io~projected/kube-api-access-qd5b8 DeviceMajor:0 DeviceMinor:228 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-272 DeviceMajor:0 DeviceMinor:272 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a8dee9b06f8070907d581c085c189097c986b3347dddd64dfe1637cd79b4352c/userdata/shm DeviceMajor:0 DeviceMinor:63 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/238ff7d8-7da8-4cd8-9d45-deedf615211e/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:48 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:201 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3f97ee34-5dfc-454e-ba5c-afd939f3bd25/volumes/kubernetes.io~projected/kube-api-access-cdtc5 DeviceMajor:0 DeviceMinor:209 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-284 DeviceMajor:0 DeviceMinor:284 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-354 DeviceMajor:0 DeviceMinor:354 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a32aacdc-53be-41cf-839e-435d43361890/volumes/kubernetes.io~projected/kube-api-access-424gm DeviceMajor:0 DeviceMinor:210 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5e74071e-0462-42b5-a05c-d90bb638478a/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:219 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-339 DeviceMajor:0 DeviceMinor:339 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ed08bc51-70e6-4658-adb3-ed36f5176552/volumes/kubernetes.io~projected/kube-api-access-pd6bv DeviceMajor:0 DeviceMinor:52 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e369a0f022d80f14a70bcfff6d7599bae3b42ee10574fb6f745e5f9bcc7fa2ad/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:087dbdeaadb724f MacAddress:36:76:63:48:3e:40 Speed:10000 Mtu:8901} {Name:4b41219f99fde72 MacAddress:ae:d7:bb:ba:b7:6a Speed:10000 Mtu:8901} {Name:627a6b6e27f45f5 MacAddress:46:92:05:0f:ea:c1 Speed:10000 Mtu:8901} {Name:65c7594af33d218 MacAddress:c2:62:fe:90:7e:e2 Speed:10000 Mtu:8901} {Name:6fc99d3eb77d112 MacAddress:32:c0:01:62:26:35 Speed:10000 Mtu:8901} {Name:7429a4fe42cd3da MacAddress:9e:04:ce:e9:8c:25 Speed:10000 Mtu:8901} {Name:75e0ccc2236e8a6 MacAddress:f2:0a:98:85:13:39 Speed:10000 Mtu:8901} {Name:82a9465e06fce6f MacAddress:be:66:3b:3e:10:43 Speed:10000 Mtu:8901} {Name:8da43005c1309e0 MacAddress:9a:6e:a3:9c:47:71 Speed:10000 Mtu:8901} {Name:a55e6c85fda4723 MacAddress:72:b7:b0:d8:d8:1e Speed:10000 Mtu:8901} {Name:b1588b1f5ed94ad MacAddress:2e:e6:7c:81:97:e0 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:06:ff:e3:9e:ab:e3 Speed:0 Mtu:9001} {Name:br-int MacAddress:8e:de:bd:e6:5b:57 Speed:0 Mtu:8901} {Name:c20b0427474a203 MacAddress:26:fe:69:4f:1b:c9 Speed:10000 Mtu:8901} {Name:c25682648a4da36 MacAddress:3a:a0:28:ca:90:ee Speed:10000 Mtu:8901} {Name:c5dd861e453dd12 MacAddress:aa:3c:2b:8a:e9:9c Speed:10000 Mtu:8901} {Name:d6179def9b341a1 MacAddress:fe:69:a4:dd:05:45 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:06:ff:e3:9e:ab:e3 Speed:0 Mtu:9001} {Name:fd3c3fb68c3b02c MacAddress:3a:e0:40:87:d9:79 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:6e:bb:24:59:58:ed Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:2e:d0:97:1b:9b:03 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 21:42:26.471327 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.459817    5061 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 21:42:26.471327 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.459903    5061 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 21:42:26.471327 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.460235    5061 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 21:42:26.471327 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.460272    5061 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-60.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 21:42:26.471327 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.460463    5061 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 21:42:26.471327 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.460475    5061 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 21:42:26.471327 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.460488    5061 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 21:42:26.471327 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.460514    5061 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 21:42:26.471327 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.460632    5061 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 21:42:26.471327 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.460844    5061 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 21:42:26.471327 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.460969    5061 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 21:42:26.471327 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.461009    5061 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 21:42:26.471327 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.461021    5061 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 21:42:26.471327 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.461035    5061 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 21:42:26.471327 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.461049    5061 kubelet.go:405] "Adding apiserver pod source"
Oct 30 21:42:26.471327 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.461059    5061 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 21:42:26.471327 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.462396    5061 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 21:42:26.471327 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.462409    5061 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 21:42:26.471327 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.462961    5061 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 21:42:26.471327 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.465585    5061 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 21:42:26.471327 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.465595    5061 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 21:42:26.471327 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.465696    5061 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 30 21:42:26.471327 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.465895    5061 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 21:42:26.471327 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.465924    5061 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 21:42:26.471327 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.466029    5061 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 21:42:26.471327 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.466042    5061 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 21:42:26.471327 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.466049    5061 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 21:42:26.471327 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.466054    5061 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 21:42:26.471327 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.466063    5061 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 21:42:26.471327 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.466073    5061 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 21:42:26.471327 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.466090    5061 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 21:42:26.471327 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.466097    5061 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 21:42:26.471327 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.466103    5061 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 21:42:26.471327 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.466111    5061 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 21:42:26.471327 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.466124    5061 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 21:42:26.471327 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.466133    5061 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 21:42:26.471327 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.466159    5061 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 21:42:26.471327 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.466168    5061 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 21:42:26.471327 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.469709    5061 server.go:1263] "Started kubelet"
Oct 30 21:42:26.471327 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.469903    5061 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 21:42:26.478859 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.478802    5061 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 21:42:26.479016 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.479004    5061 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 21:42:26.479576 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.479558    5061 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 21:42:26.480292 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.479992    5061 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 21:42:26.480455 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.480436    5061 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 21:42:26.480921 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.480872    5061 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 21:36:46 +0000 UTC" deadline="2028-05-27 10:48:36.839102526 +0000 UTC"
Oct 30 21:42:26.480921 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.480917    5061 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13789h6m10.358189339s"
Oct 30 21:42:26.481122 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.481108    5061 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 21:42:26.481168 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.481122    5061 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 21:42:26.481238 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.481227    5061 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 21:42:26.481971 ip-10-0-130-60 kubenswrapper[5061]: E1030 21:42:26.481945    5061 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-60.us-west-2.compute.internal\" not found"
Oct 30 21:42:26.483053 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.483038    5061 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 21:42:26.486677 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.486643    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ed08bc51-70e6-4658-adb3-ed36f5176552" volumeName="kubernetes.io/empty-dir/ed08bc51-70e6-4658-adb3-ed36f5176552-etc-tuned" seLinuxMountContext=""
Oct 30 21:42:26.486775 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.486682    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="538d5103-1c92-48f1-acc7-f90956848727" volumeName="kubernetes.io/configmap/538d5103-1c92-48f1-acc7-f90956848727-ovnkube-config" seLinuxMountContext=""
Oct 30 21:42:26.486775 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.486698    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="613e7c8a-e2c7-424a-93e0-7fa86d18a8a9" volumeName="kubernetes.io/secret/613e7c8a-e2c7-424a-93e0-7fa86d18a8a9-installation-pull-secrets" seLinuxMountContext=""
Oct 30 21:42:26.486775 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.486710    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="798f248f-d429-4f56-8752-e157dd94206b" volumeName="kubernetes.io/projected/798f248f-d429-4f56-8752-e157dd94206b-kube-api-access-8w7g2" seLinuxMountContext=""
Oct 30 21:42:26.486775 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.486728    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c69312da-8be3-4931-8a81-a8ff3f86d5e6" volumeName="kubernetes.io/configmap/c69312da-8be3-4931-8a81-a8ff3f86d5e6-service-ca-bundle" seLinuxMountContext=""
Oct 30 21:42:26.486775 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.486739    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c69312da-8be3-4931-8a81-a8ff3f86d5e6" volumeName="kubernetes.io/projected/c69312da-8be3-4931-8a81-a8ff3f86d5e6-kube-api-access-qd5b8" seLinuxMountContext=""
Oct 30 21:42:26.486775 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.486751    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c69312da-8be3-4931-8a81-a8ff3f86d5e6" volumeName="kubernetes.io/secret/c69312da-8be3-4931-8a81-a8ff3f86d5e6-serving-cert" seLinuxMountContext=""
Oct 30 21:42:26.486775 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.486763    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cad67bc5-c8d2-497c-ba1b-cdbae5822cfa" volumeName="kubernetes.io/configmap/cad67bc5-c8d2-497c-ba1b-cdbae5822cfa-trusted-ca" seLinuxMountContext=""
Oct 30 21:42:26.487076 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.486778    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="033356b5-3c05-4f49-ad6a-7c369259970d" volumeName="kubernetes.io/projected/033356b5-3c05-4f49-ad6a-7c369259970d-kube-api-access-d9grk" seLinuxMountContext=""
Oct 30 21:42:26.487076 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.486796    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="28210ad9-18f0-4ecf-9bfc-b256440f0ce8" volumeName="kubernetes.io/secret/28210ad9-18f0-4ecf-9bfc-b256440f0ce8-cert" seLinuxMountContext=""
Oct 30 21:42:26.487076 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.486812    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b227ad6a-9e59-415e-b7e6-f7c25e988abb" volumeName="kubernetes.io/projected/b227ad6a-9e59-415e-b7e6-f7c25e988abb-kube-api-access-qpbtv" seLinuxMountContext=""
Oct 30 21:42:26.487076 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.486825    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c798fd57-e0be-4b61-860b-0b833aca7784" volumeName="kubernetes.io/projected/c798fd57-e0be-4b61-860b-0b833aca7784-kube-api-access-xk5zb" seLinuxMountContext=""
Oct 30 21:42:26.487076 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.486836    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0bb51175-fa5e-4472-8d83-106bca721585" volumeName="kubernetes.io/projected/0bb51175-fa5e-4472-8d83-106bca721585-kube-api-access-pdbmm" seLinuxMountContext=""
Oct 30 21:42:26.487076 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.486847    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1f3b6a23-d7ea-4ce4-b1e8-68133c4b4ebc" volumeName="kubernetes.io/configmap/1f3b6a23-d7ea-4ce4-b1e8-68133c4b4ebc-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 30 21:42:26.487076 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.486861    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="538d5103-1c92-48f1-acc7-f90956848727" volumeName="kubernetes.io/configmap/538d5103-1c92-48f1-acc7-f90956848727-ovnkube-script-lib" seLinuxMountContext=""
Oct 30 21:42:26.487076 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.486872    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d" volumeName="kubernetes.io/projected/9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d-kube-api-access-n4tb7" seLinuxMountContext=""
Oct 30 21:42:26.487076 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.486885    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d" volumeName="kubernetes.io/secret/9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d-stats-auth" seLinuxMountContext=""
Oct 30 21:42:26.487076 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.486896    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b227ad6a-9e59-415e-b7e6-f7c25e988abb" volumeName="kubernetes.io/configmap/b227ad6a-9e59-415e-b7e6-f7c25e988abb-serviceca" seLinuxMountContext=""
Oct 30 21:42:26.487076 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.486907    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c69312da-8be3-4931-8a81-a8ff3f86d5e6" volumeName="kubernetes.io/configmap/c69312da-8be3-4931-8a81-a8ff3f86d5e6-trusted-ca-bundle" seLinuxMountContext=""
Oct 30 21:42:26.487076 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.486918    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cad67bc5-c8d2-497c-ba1b-cdbae5822cfa" volumeName="kubernetes.io/projected/cad67bc5-c8d2-497c-ba1b-cdbae5822cfa-bound-sa-token" seLinuxMountContext=""
Oct 30 21:42:26.487076 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.486929    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3f97ee34-5dfc-454e-ba5c-afd939f3bd25" volumeName="kubernetes.io/projected/3f97ee34-5dfc-454e-ba5c-afd939f3bd25-kube-api-access-cdtc5" seLinuxMountContext=""
Oct 30 21:42:26.487076 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.486940    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="102b8049-7349-4e2b-974a-44c901eebcf3" volumeName="kubernetes.io/configmap/102b8049-7349-4e2b-974a-44c901eebcf3-telemetry-config" seLinuxMountContext=""
Oct 30 21:42:26.487076 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.486954    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1f3b6a23-d7ea-4ce4-b1e8-68133c4b4ebc" volumeName="kubernetes.io/configmap/1f3b6a23-d7ea-4ce4-b1e8-68133c4b4ebc-cni-binary-copy" seLinuxMountContext=""
Oct 30 21:42:26.487076 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.486965    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="238ff7d8-7da8-4cd8-9d45-deedf615211e" volumeName="kubernetes.io/configmap/238ff7d8-7da8-4cd8-9d45-deedf615211e-konnectivity-ca" seLinuxMountContext=""
Oct 30 21:42:26.487076 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.486976    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5e74071e-0462-42b5-a05c-d90bb638478a" volumeName="kubernetes.io/secret/5e74071e-0462-42b5-a05c-d90bb638478a-serving-cert" seLinuxMountContext=""
Oct 30 21:42:26.487076 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.486991    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d700d1eb-fbf4-45df-9b57-eca08d87ebe2" volumeName="kubernetes.io/projected/d700d1eb-fbf4-45df-9b57-eca08d87ebe2-kube-api-access-nhbrj" seLinuxMountContext=""
Oct 30 21:42:26.487076 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.487002    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ed08bc51-70e6-4658-adb3-ed36f5176552" volumeName="kubernetes.io/projected/ed08bc51-70e6-4658-adb3-ed36f5176552-kube-api-access-pd6bv" seLinuxMountContext=""
Oct 30 21:42:26.487076 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.487014    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0126a0a0-a1c8-4db3-a6af-629360e247f4" volumeName="kubernetes.io/projected/0126a0a0-a1c8-4db3-a6af-629360e247f4-kube-api-access-srwt7" seLinuxMountContext=""
Oct 30 21:42:26.487076 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.487031    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0bb51175-fa5e-4472-8d83-106bca721585" volumeName="kubernetes.io/configmap/0bb51175-fa5e-4472-8d83-106bca721585-samples-operator-config" seLinuxMountContext=""
Oct 30 21:42:26.487076 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.487042    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="101c697b-d7b8-4cfe-9d4c-818b5c04108f" volumeName="kubernetes.io/empty-dir/101c697b-d7b8-4cfe-9d4c-818b5c04108f-archives-path" seLinuxMountContext=""
Oct 30 21:42:26.487076 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.487053    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="102b8049-7349-4e2b-974a-44c901eebcf3" volumeName="kubernetes.io/projected/102b8049-7349-4e2b-974a-44c901eebcf3-kube-api-access-5fcz7" seLinuxMountContext=""
Oct 30 21:42:26.487076 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.487064    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5e74071e-0462-42b5-a05c-d90bb638478a" volumeName="kubernetes.io/projected/5e74071e-0462-42b5-a05c-d90bb638478a-kube-api-access-vqmr4" seLinuxMountContext=""
Oct 30 21:42:26.487076 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.487076    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="613e7c8a-e2c7-424a-93e0-7fa86d18a8a9" volumeName="kubernetes.io/empty-dir/613e7c8a-e2c7-424a-93e0-7fa86d18a8a9-ca-trust-extracted" seLinuxMountContext=""
Oct 30 21:42:26.487076 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.487087    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="798f248f-d429-4f56-8752-e157dd94206b" volumeName="kubernetes.io/secret/798f248f-d429-4f56-8752-e157dd94206b-serving-cert" seLinuxMountContext=""
Oct 30 21:42:26.488169 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.487099    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0929b9f6-12ec-4f4d-8ba3-5cef9defba15" volumeName="kubernetes.io/projected/0929b9f6-12ec-4f4d-8ba3-5cef9defba15-kube-api-access-swjcw" seLinuxMountContext=""
Oct 30 21:42:26.488169 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.487110    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="84d9b682-3241-40af-87d5-f87d54ce3697" volumeName="kubernetes.io/empty-dir/84d9b682-3241-40af-87d5-f87d54ce3697-tmp-dir" seLinuxMountContext=""
Oct 30 21:42:26.488169 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.487121    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a2e3b25e-b6dc-46ef-86e8-4e70f9dad200" volumeName="kubernetes.io/configmap/a2e3b25e-b6dc-46ef-86e8-4e70f9dad200-cni-binary-copy" seLinuxMountContext=""
Oct 30 21:42:26.488169 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.487133    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a32aacdc-53be-41cf-839e-435d43361890" volumeName="kubernetes.io/projected/a32aacdc-53be-41cf-839e-435d43361890-kube-api-access-424gm" seLinuxMountContext=""
Oct 30 21:42:26.488169 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.487143    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c2386477-903e-4c6c-b807-3f473a58a6b6" volumeName="kubernetes.io/configmap/c2386477-903e-4c6c-b807-3f473a58a6b6-signing-cabundle" seLinuxMountContext=""
Oct 30 21:42:26.488169 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.487153    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c2386477-903e-4c6c-b807-3f473a58a6b6" volumeName="kubernetes.io/projected/c2386477-903e-4c6c-b807-3f473a58a6b6-kube-api-access-zv2qw" seLinuxMountContext=""
Oct 30 21:42:26.488169 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.487164    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cad67bc5-c8d2-497c-ba1b-cdbae5822cfa" volumeName="kubernetes.io/projected/cad67bc5-c8d2-497c-ba1b-cdbae5822cfa-kube-api-access-m9dpv" seLinuxMountContext=""
Oct 30 21:42:26.488169 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.487175    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cad67bc5-c8d2-497c-ba1b-cdbae5822cfa" volumeName="kubernetes.io/secret/cad67bc5-c8d2-497c-ba1b-cdbae5822cfa-image-registry-private-configuration" seLinuxMountContext=""
Oct 30 21:42:26.488169 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.487186    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0126a0a0-a1c8-4db3-a6af-629360e247f4" volumeName="kubernetes.io/configmap/0126a0a0-a1c8-4db3-a6af-629360e247f4-config" seLinuxMountContext=""
Oct 30 21:42:26.488169 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.487197    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="613e7c8a-e2c7-424a-93e0-7fa86d18a8a9" volumeName="kubernetes.io/configmap/613e7c8a-e2c7-424a-93e0-7fa86d18a8a9-registry-certificates" seLinuxMountContext=""
Oct 30 21:42:26.488169 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.487209    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cad67bc5-c8d2-497c-ba1b-cdbae5822cfa" volumeName="kubernetes.io/empty-dir/cad67bc5-c8d2-497c-ba1b-cdbae5822cfa-ca-trust-extracted" seLinuxMountContext=""
Oct 30 21:42:26.488169 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.487219    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cad67bc5-c8d2-497c-ba1b-cdbae5822cfa" volumeName="kubernetes.io/secret/cad67bc5-c8d2-497c-ba1b-cdbae5822cfa-installation-pull-secrets" seLinuxMountContext=""
Oct 30 21:42:26.488169 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.487231    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="84d9b682-3241-40af-87d5-f87d54ce3697" volumeName="kubernetes.io/projected/84d9b682-3241-40af-87d5-f87d54ce3697-kube-api-access-mms6q" seLinuxMountContext=""
Oct 30 21:42:26.488169 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.487242    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="879022d4-06b9-44cc-a2a2-dc307b27345e" volumeName="kubernetes.io/projected/879022d4-06b9-44cc-a2a2-dc307b27345e-kube-api-access-4658n" seLinuxMountContext=""
Oct 30 21:42:26.488169 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.487254    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2205e8e6-f038-47fa-b393-0abe4d622e03" volumeName="kubernetes.io/projected/2205e8e6-f038-47fa-b393-0abe4d622e03-kube-api-access-rbwzr" seLinuxMountContext=""
Oct 30 21:42:26.488169 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.487265    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="28210ad9-18f0-4ecf-9bfc-b256440f0ce8" volumeName="kubernetes.io/projected/28210ad9-18f0-4ecf-9bfc-b256440f0ce8-kube-api-access-l75p7" seLinuxMountContext=""
Oct 30 21:42:26.488169 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.487276    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5e74071e-0462-42b5-a05c-d90bb638478a" volumeName="kubernetes.io/configmap/5e74071e-0462-42b5-a05c-d90bb638478a-config" seLinuxMountContext=""
Oct 30 21:42:26.488169 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.487287    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="613e7c8a-e2c7-424a-93e0-7fa86d18a8a9" volumeName="kubernetes.io/projected/613e7c8a-e2c7-424a-93e0-7fa86d18a8a9-bound-sa-token" seLinuxMountContext=""
Oct 30 21:42:26.488169 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.487300    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="101c697b-d7b8-4cfe-9d4c-818b5c04108f" volumeName="kubernetes.io/configmap/101c697b-d7b8-4cfe-9d4c-818b5c04108f-service-ca-bundle" seLinuxMountContext=""
Oct 30 21:42:26.488169 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.487310    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="613e7c8a-e2c7-424a-93e0-7fa86d18a8a9" volumeName="kubernetes.io/configmap/613e7c8a-e2c7-424a-93e0-7fa86d18a8a9-trusted-ca" seLinuxMountContext=""
Oct 30 21:42:26.488169 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.487321    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="71ddf455-cbf7-49a6-9773-30f7e4997a39" volumeName="kubernetes.io/projected/71ddf455-cbf7-49a6-9773-30f7e4997a39-kube-api-access-7hkfh" seLinuxMountContext=""
Oct 30 21:42:26.488169 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.487332    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1f3b6a23-d7ea-4ce4-b1e8-68133c4b4ebc" volumeName="kubernetes.io/projected/1f3b6a23-d7ea-4ce4-b1e8-68133c4b4ebc-kube-api-access-7gq9m" seLinuxMountContext=""
Oct 30 21:42:26.488169 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.487351    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d" volumeName="kubernetes.io/secret/9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d-default-certificate" seLinuxMountContext=""
Oct 30 21:42:26.488169 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.487362    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c69312da-8be3-4931-8a81-a8ff3f86d5e6" volumeName="kubernetes.io/empty-dir/c69312da-8be3-4931-8a81-a8ff3f86d5e6-snapshots" seLinuxMountContext=""
Oct 30 21:42:26.488169 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.487373    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c69312da-8be3-4931-8a81-a8ff3f86d5e6" volumeName="kubernetes.io/empty-dir/c69312da-8be3-4931-8a81-a8ff3f86d5e6-tmp" seLinuxMountContext=""
Oct 30 21:42:26.488169 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.487384    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cad67bc5-c8d2-497c-ba1b-cdbae5822cfa" volumeName="kubernetes.io/configmap/cad67bc5-c8d2-497c-ba1b-cdbae5822cfa-registry-certificates" seLinuxMountContext=""
Oct 30 21:42:26.488169 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.487395    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="238ff7d8-7da8-4cd8-9d45-deedf615211e" volumeName="kubernetes.io/secret/238ff7d8-7da8-4cd8-9d45-deedf615211e-agent-certs" seLinuxMountContext=""
Oct 30 21:42:26.488169 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.487405    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="538d5103-1c92-48f1-acc7-f90956848727" volumeName="kubernetes.io/projected/538d5103-1c92-48f1-acc7-f90956848727-kube-api-access-hrpz5" seLinuxMountContext=""
Oct 30 21:42:26.488169 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.487417    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="798f248f-d429-4f56-8752-e157dd94206b" volumeName="kubernetes.io/configmap/798f248f-d429-4f56-8752-e157dd94206b-config" seLinuxMountContext=""
Oct 30 21:42:26.488169 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.487427    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="84d9b682-3241-40af-87d5-f87d54ce3697" volumeName="kubernetes.io/configmap/84d9b682-3241-40af-87d5-f87d54ce3697-config-volume" seLinuxMountContext=""
Oct 30 21:42:26.488169 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.487439    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c2386477-903e-4c6c-b807-3f473a58a6b6" volumeName="kubernetes.io/secret/c2386477-903e-4c6c-b807-3f473a58a6b6-signing-key" seLinuxMountContext=""
Oct 30 21:42:26.488169 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.487452    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3784e680-5efe-4ffb-ad71-956d05b57dce" volumeName="kubernetes.io/projected/3784e680-5efe-4ffb-ad71-956d05b57dce-kube-api-access-579v9" seLinuxMountContext=""
Oct 30 21:42:26.488169 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.487463    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="538d5103-1c92-48f1-acc7-f90956848727" volumeName="kubernetes.io/configmap/538d5103-1c92-48f1-acc7-f90956848727-env-overrides" seLinuxMountContext=""
Oct 30 21:42:26.488169 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.487474    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="613e7c8a-e2c7-424a-93e0-7fa86d18a8a9" volumeName="kubernetes.io/secret/613e7c8a-e2c7-424a-93e0-7fa86d18a8a9-image-registry-private-configuration" seLinuxMountContext=""
Oct 30 21:42:26.488169 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.487485    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ed08bc51-70e6-4658-adb3-ed36f5176552" volumeName="kubernetes.io/empty-dir/ed08bc51-70e6-4658-adb3-ed36f5176552-tmp" seLinuxMountContext=""
Oct 30 21:42:26.488169 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.487496    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0126a0a0-a1c8-4db3-a6af-629360e247f4" volumeName="kubernetes.io/secret/0126a0a0-a1c8-4db3-a6af-629360e247f4-serving-cert" seLinuxMountContext=""
Oct 30 21:42:26.488169 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.487508    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="101c697b-d7b8-4cfe-9d4c-818b5c04108f" volumeName="kubernetes.io/configmap/101c697b-d7b8-4cfe-9d4c-818b5c04108f-trusted-ca-bundle" seLinuxMountContext=""
Oct 30 21:42:26.488169 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.487519    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1f3b6a23-d7ea-4ce4-b1e8-68133c4b4ebc" volumeName="kubernetes.io/configmap/1f3b6a23-d7ea-4ce4-b1e8-68133c4b4ebc-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 30 21:42:26.488169 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.487545    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="74ff988a-c05e-4a3e-adff-5945b79cf248" volumeName="kubernetes.io/empty-dir/74ff988a-c05e-4a3e-adff-5945b79cf248-data-volume" seLinuxMountContext=""
Oct 30 21:42:26.488169 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.487846    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a2e3b25e-b6dc-46ef-86e8-4e70f9dad200" volumeName="kubernetes.io/projected/a2e3b25e-b6dc-46ef-86e8-4e70f9dad200-kube-api-access-p8b7v" seLinuxMountContext=""
Oct 30 21:42:26.488169 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.487861    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f648e933-f035-486a-88d5-1b185c89782e" volumeName="kubernetes.io/configmap/f648e933-f035-486a-88d5-1b185c89782e-nginx-conf" seLinuxMountContext=""
Oct 30 21:42:26.488169 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.487873    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4427f59d-8003-440f-9646-a46b176cc4a7" volumeName="kubernetes.io/projected/4427f59d-8003-440f-9646-a46b176cc4a7-kube-api-access-nc76p" seLinuxMountContext=""
Oct 30 21:42:26.488169 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.487886    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="538d5103-1c92-48f1-acc7-f90956848727" volumeName="kubernetes.io/secret/538d5103-1c92-48f1-acc7-f90956848727-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 30 21:42:26.488169 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.487897    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5e74071e-0462-42b5-a05c-d90bb638478a" volumeName="kubernetes.io/configmap/5e74071e-0462-42b5-a05c-d90bb638478a-trusted-ca" seLinuxMountContext=""
Oct 30 21:42:26.488169 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.487908    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="74ff988a-c05e-4a3e-adff-5945b79cf248" volumeName="kubernetes.io/projected/74ff988a-c05e-4a3e-adff-5945b79cf248-kube-api-access-67qtb" seLinuxMountContext=""
Oct 30 21:42:26.488169 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.487920    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a2e3b25e-b6dc-46ef-86e8-4e70f9dad200" volumeName="kubernetes.io/configmap/a2e3b25e-b6dc-46ef-86e8-4e70f9dad200-multus-daemon-config" seLinuxMountContext=""
Oct 30 21:42:26.488169 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.487931    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fe41c70b-7e2c-432f-8389-0282d6fdb332" volumeName="kubernetes.io/secret/fe41c70b-7e2c-432f-8389-0282d6fdb332-original-pull-secret" seLinuxMountContext=""
Oct 30 21:42:26.488169 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.487942    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="71ddf455-cbf7-49a6-9773-30f7e4997a39" volumeName="kubernetes.io/empty-dir/71ddf455-cbf7-49a6-9773-30f7e4997a39-tmp-dir" seLinuxMountContext=""
Oct 30 21:42:26.488169 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.487953    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0929b9f6-12ec-4f4d-8ba3-5cef9defba15" volumeName="kubernetes.io/configmap/0929b9f6-12ec-4f4d-8ba3-5cef9defba15-iptables-alerter-script" seLinuxMountContext=""
Oct 30 21:42:26.488169 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.487964    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="101c697b-d7b8-4cfe-9d4c-818b5c04108f" volumeName="kubernetes.io/projected/101c697b-d7b8-4cfe-9d4c-818b5c04108f-kube-api-access-xm5l5" seLinuxMountContext=""
Oct 30 21:42:26.488169 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.487975    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="20f55ce3-5b04-4916-b529-0dd596aba68f" volumeName="kubernetes.io/projected/20f55ce3-5b04-4916-b529-0dd596aba68f-kube-api-access-kr7ww" seLinuxMountContext=""
Oct 30 21:42:26.488169 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.487988    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="613e7c8a-e2c7-424a-93e0-7fa86d18a8a9" volumeName="kubernetes.io/projected/613e7c8a-e2c7-424a-93e0-7fa86d18a8a9-kube-api-access-hw226" seLinuxMountContext=""
Oct 30 21:42:26.488169 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.488000    5061 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 21:42:26.488169 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.488008    5061 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 21:42:26.490480 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.488896    5061 factory.go:55] Registering systemd factory
Oct 30 21:42:26.490480 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.488910    5061 factory.go:223] Registration of the systemd container factory successfully
Oct 30 21:42:26.490480 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.489174    5061 factory.go:153] Registering CRI-O factory
Oct 30 21:42:26.490480 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.489184    5061 factory.go:223] Registration of the crio container factory successfully
Oct 30 21:42:26.490480 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.489226    5061 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 21:42:26.490480 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.489242    5061 factory.go:103] Registering Raw factory
Oct 30 21:42:26.490480 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.489255    5061 manager.go:1199] Started watching for new ooms in manager
Oct 30 21:42:26.490480 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.489580    5061 manager.go:319] Starting recovery of all containers
Oct 30 21:42:26.501177 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.501153    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-sclrt_5e74071e-0462-42b5-a05c-d90bb638478a/console-operator/1.log"
Oct 30 21:42:26.501266 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.501174    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-sclrt_5e74071e-0462-42b5-a05c-d90bb638478a/console-operator/1.log"
Oct 30 21:42:26.518563 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.518488    5061 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 21:42:26.520188 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.520165    5061 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 21:42:26.520188 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.520190    5061 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 21:42:26.520340 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.520209    5061 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 21:42:26.520340 ip-10-0-130-60 kubenswrapper[5061]: E1030 21:42:26.520244    5061 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 30 21:42:26.546751 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.546713    5061 generic.go:356] "Generic (PLEG): container finished" podID="cf62eb3c9c04acf0f549e44a10220f5a" containerID="25f6c7e10a3841baf6120fa239857b451f78ee39a794830fab2f601a27aad210" exitCode=0
Oct 30 21:42:26.563518 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.563492    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-sclrt_5e74071e-0462-42b5-a05c-d90bb638478a/console-operator/1.log"
Oct 30 21:42:26.563518 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.563544    5061 generic.go:356] "Generic (PLEG): container finished" podID="5e74071e-0462-42b5-a05c-d90bb638478a" containerID="fef0614d3ea43c5da061d6b88ea8aa29855cf27a2684dc399ca63400506200bb" exitCode=255
Oct 30 21:42:26.580681 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.580648    5061 generic.go:356] "Generic (PLEG): container finished" podID="1f3b6a23-d7ea-4ce4-b1e8-68133c4b4ebc" containerID="4d2c6ac7e5209c068013758686fb1e2f2292f9fda8333cf1f97cf491ae8201b6" exitCode=0
Oct 30 21:42:26.580800 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.580684    5061 generic.go:356] "Generic (PLEG): container finished" podID="1f3b6a23-d7ea-4ce4-b1e8-68133c4b4ebc" containerID="84b653b15569d0c3ddcec103af4165ac52ce183d9e0ef60757af9badc5079ff7" exitCode=0
Oct 30 21:42:26.580800 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.580698    5061 generic.go:356] "Generic (PLEG): container finished" podID="1f3b6a23-d7ea-4ce4-b1e8-68133c4b4ebc" containerID="f66cf20a6bc74068121170fff522486d3cfe02a8788157ebc69dea717abeb96e" exitCode=0
Oct 30 21:42:26.580800 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.580707    5061 generic.go:356] "Generic (PLEG): container finished" podID="1f3b6a23-d7ea-4ce4-b1e8-68133c4b4ebc" containerID="df74a38c1973f552b2f34ba504e4288306b7f3398f80f0068f4d63233c397d8f" exitCode=0
Oct 30 21:42:26.580800 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.580720    5061 generic.go:356] "Generic (PLEG): container finished" podID="1f3b6a23-d7ea-4ce4-b1e8-68133c4b4ebc" containerID="5270a081e0c82cc2f7477dbab248e9aedcf934fbdcd6e5ae2af31fb686e18781" exitCode=0
Oct 30 21:42:26.580800 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.580737    5061 generic.go:356] "Generic (PLEG): container finished" podID="1f3b6a23-d7ea-4ce4-b1e8-68133c4b4ebc" containerID="9c3b30ec66ad103cdf19bbf59344440ab5f9f4677c7d202a77f05fb3a1191a22" exitCode=0
Oct 30 21:42:26.582060 ip-10-0-130-60 kubenswrapper[5061]: E1030 21:42:26.582037    5061 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-60.us-west-2.compute.internal\" not found"
Oct 30 21:42:26.604225 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.604207    5061 manager.go:324] Recovery completed
Oct 30 21:42:26.615122 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.615102    5061 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:42:26.620412 ip-10-0-130-60 kubenswrapper[5061]: E1030 21:42:26.620383    5061 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Oct 30 21:42:26.628785 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.628768    5061 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:42:26.631001 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.630977    5061 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-60.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:42:26.631088 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.631015    5061 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-60.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:42:26.631088 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.631039    5061 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-60.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:42:26.636212 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.636192    5061 cpu_manager.go:226] "Starting" policy="none"
Oct 30 21:42:26.636212 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.636208    5061 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 21:42:26.636367 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.636228    5061 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 21:42:26.636367 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.636241    5061 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:42:26.636367 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.636363    5061 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 30 21:42:26.636480 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.636377    5061 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 30 21:42:26.636480 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.636399    5061 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 30 21:42:26.636480 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.636407    5061 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 30 21:42:26.636480 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.636414    5061 policy_none.go:50] "Start"
Oct 30 21:42:26.636480 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.636425    5061 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 21:42:26.636480 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.636436    5061 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 21:42:26.636663 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.636613    5061 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 30 21:42:26.636663 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.636623    5061 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 30 21:42:26.636663 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.636631    5061 policy_none.go:44] "Start"
Oct 30 21:42:26.637240 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.637219    5061 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-60.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:42:26.637318 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.637254    5061 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-60.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:42:26.637318 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.637268    5061 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-60.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:42:26.642391 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.642362    5061 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 21:42:26.649065 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.645239    5061 manager.go:341] "Starting Device Plugin manager"
Oct 30 21:42:26.649065 ip-10-0-130-60 kubenswrapper[5061]: E1030 21:42:26.645278    5061 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 21:42:26.649065 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.645291    5061 server.go:87] "Starting device plugin registration server"
Oct 30 21:42:26.649065 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.646681    5061 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 21:42:26.649065 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.646694    5061 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 21:42:26.649065 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.647453    5061 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 21:42:26.649065 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.647564    5061 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 21:42:26.649065 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.647598    5061 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 21:42:26.649065 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.647605    5061 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 21:42:26.649065 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.647659    5061 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T21:42:26.647589108Z","UUID":"6afee904-819e-4cd6-8165-bb9042a05c9f","Handler":null,"Name":"","Endpoint":""}
Oct 30 21:42:26.649433 ip-10-0-130-60 kubenswrapper[5061]: E1030 21:42:26.649082    5061 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 21:42:26.649433 ip-10-0-130-60 kubenswrapper[5061]: E1030 21:42:26.649110    5061 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-60.us-west-2.compute.internal\" not found"
Oct 30 21:42:26.649433 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.649212    5061 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 21:42:26.649433 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.649231    5061 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 21:42:26.660317 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.660301    5061 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:42:26.661808 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.661790    5061 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:42:26.709780 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.709731    5061 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:42:26.709848 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.709833    5061 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 30 21:42:26.748382 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.748362    5061 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:42:26.749329 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.749313    5061 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-60.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:42:26.749396 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.749342    5061 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-60.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:42:26.749396 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.749354    5061 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-60.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:42:26.749396 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.749393    5061 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-130-60.us-west-2.compute.internal"
Oct 30 21:42:26.820873 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.820830    5061 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-130-60.us-west-2.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-60.us-west-2.compute.internal"]
Oct 30 21:42:26.821449 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.821435    5061 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-60.us-west-2.compute.internal"
Oct 30 21:42:26.821518 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.821488    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-60.us-west-2.compute.internal" event={"ID":"c0b65115e7be9fc3f67a9c81d1477e73","Type":"ContainerStarted","Data":"d1d30500380aa168c8228952650114a5604f5ca3e2c4d115b5dc145ee58e92e6"}
Oct 30 21:42:26.821580 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.821545    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-60.us-west-2.compute.internal" event={"ID":"c0b65115e7be9fc3f67a9c81d1477e73","Type":"ContainerStarted","Data":"23d53277e0647a0a05ba276cdb89af8108237e92f2dae2b39a5e3f45d1452f89"}
Oct 30 21:42:26.821580 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.821565    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-60.us-west-2.compute.internal" event={"ID":"cf62eb3c9c04acf0f549e44a10220f5a","Type":"ContainerStarted","Data":"5c04ec9c260142a34a460c97120a73f230417cb57d760dab3a98d0b1da29c2d2"}
Oct 30 21:42:26.821580 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.821578    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-60.us-west-2.compute.internal" event={"ID":"cf62eb3c9c04acf0f549e44a10220f5a","Type":"ContainerDied","Data":"25f6c7e10a3841baf6120fa239857b451f78ee39a794830fab2f601a27aad210"}
Oct 30 21:42:26.821691 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.821591    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-60.us-west-2.compute.internal" event={"ID":"cf62eb3c9c04acf0f549e44a10220f5a","Type":"ContainerStarted","Data":"76ef14d66ab9af8383c8ea83abcbfb7325220d8b7e326ef09c6a16fab7db51b2"}
Oct 30 21:42:26.821828 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.821815    5061 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-60.us-west-2.compute.internal"
Oct 30 21:42:26.872488 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.872469    5061 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-130-60.us-west-2.compute.internal"
Oct 30 21:42:26.872610 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.872563    5061 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-130-60.us-west-2.compute.internal"
Oct 30 21:42:26.896535 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.896505    5061 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 21:42:26.896630 ip-10-0-130-60 kubenswrapper[5061]: E1030 21:42:26.896548    5061 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-130-60.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-60.us-west-2.compute.internal"
Oct 30 21:42:26.896775 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.896759    5061 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 21:42:26.896825 ip-10-0-130-60 kubenswrapper[5061]: E1030 21:42:26.896795    5061 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-130-60.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-60.us-west-2.compute.internal"
Oct 30 21:42:26.910981 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.910927    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cf62eb3c9c04acf0f549e44a10220f5a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-60.us-west-2.compute.internal\" (UID: \"cf62eb3c9c04acf0f549e44a10220f5a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-60.us-west-2.compute.internal"
Oct 30 21:42:26.911100 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.910986    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c0b65115e7be9fc3f67a9c81d1477e73-config\") pod \"kube-apiserver-proxy-ip-10-0-130-60.us-west-2.compute.internal\" (UID: \"c0b65115e7be9fc3f67a9c81d1477e73\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-60.us-west-2.compute.internal"
Oct 30 21:42:26.911100 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:26.911002    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cf62eb3c9c04acf0f549e44a10220f5a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-60.us-west-2.compute.internal\" (UID: \"cf62eb3c9c04acf0f549e44a10220f5a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-60.us-west-2.compute.internal"
Oct 30 21:42:27.011285 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.011211    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c0b65115e7be9fc3f67a9c81d1477e73-config\") pod \"kube-apiserver-proxy-ip-10-0-130-60.us-west-2.compute.internal\" (UID: \"c0b65115e7be9fc3f67a9c81d1477e73\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-60.us-west-2.compute.internal"
Oct 30 21:42:27.011285 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.011265    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cf62eb3c9c04acf0f549e44a10220f5a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-60.us-west-2.compute.internal\" (UID: \"cf62eb3c9c04acf0f549e44a10220f5a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-60.us-west-2.compute.internal"
Oct 30 21:42:27.011285 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.011286    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cf62eb3c9c04acf0f549e44a10220f5a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-60.us-west-2.compute.internal\" (UID: \"cf62eb3c9c04acf0f549e44a10220f5a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-60.us-west-2.compute.internal"
Oct 30 21:42:27.011492 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.011301    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c0b65115e7be9fc3f67a9c81d1477e73-config\") pod \"kube-apiserver-proxy-ip-10-0-130-60.us-west-2.compute.internal\" (UID: \"c0b65115e7be9fc3f67a9c81d1477e73\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-60.us-west-2.compute.internal"
Oct 30 21:42:27.011492 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.011322    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cf62eb3c9c04acf0f549e44a10220f5a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-60.us-west-2.compute.internal\" (UID: \"cf62eb3c9c04acf0f549e44a10220f5a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-60.us-west-2.compute.internal"
Oct 30 21:42:27.011492 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.011326    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cf62eb3c9c04acf0f549e44a10220f5a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-60.us-west-2.compute.internal\" (UID: \"cf62eb3c9c04acf0f549e44a10220f5a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-60.us-west-2.compute.internal"
Oct 30 21:42:27.462019 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.461953    5061 apiserver.go:52] "Watching apiserver"
Oct 30 21:42:27.600520 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.600496    5061 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-60.us-west-2.compute.internal"
Oct 30 21:42:27.608362 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.608336    5061 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 21:42:27.609027 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.609002    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-2wsrr","openshift-multus/network-metrics-daemon-q6tjc","openshift-console-operator/console-operator-7cd56b8dbc-sclrt","openshift-multus/multus-4nqkj","kube-system/global-pull-secret-syncer-6wbw4","openshift-multus/multus-additional-cni-plugins-b47v2","openshift-network-operator/iptables-alerter-wbq4l","kube-system/kube-apiserver-proxy-ip-10-0-130-60.us-west-2.compute.internal","openshift-cluster-node-tuning-operator/tuned-hskxk","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-tlwms","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qk9dg","openshift-image-registry/image-registry-b6cdbd45d-bdftd","openshift-ovn-kubernetes/ovnkube-node-22ldv","openshift-image-registry/image-registry-d44c8cbb7-lmw97","openshift-insights/insights-runtime-extractor-mstdj","kube-system/kas-connection-checker-685895c7c9-kz6dl","openshift-dns/dns-default-6pttx","kube-system/kas-connection-checker-685895c7c9-bgnkz","openshift-image-registry/node-ca-x8gnj","kube-system/konnectivity-agent-sztmx","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-s6427","openshift-monitoring/cluster-monitoring-operator-884576c57-ns759","openshift-service-ca-operator/service-ca-operator-544d749c8c-vqwlt","openshift-network-diagnostics/network-check-source-6f78d59f99-rmpxf","openshift-service-ca/service-ca-5fffc97bdf-th574","openshift-insights/periodic-gathering-7t4kc-b65p7","openshift-kube-storage-version-migrator/migrator-66df5d496b-q4khr","openshift-ingress-canary/ingress-canary-pzpgw","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-60.us-west-2.compute.internal","openshift-kube-storage-version-migrator/migrator-66df5d496b-rznzd","openshift-network-console/networking-console-plugin-86c478bb88-j5xsr","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-mgxcj","openshift-ingress/router-default-6c7f96b45-4wj99","openshift-network-diagnostics/network-check-target-979r9","kube-system/kas-connection-checker-685895c7c9-cklcw","openshift-insights/insights-operator-7c4986d449-l9b9d"]
Oct 30 21:42:27.610120 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.610100    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ns759"
Oct 30 21:42:27.610954 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.610932    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6c7f96b45-4wj99"
Oct 30 21:42:27.611360 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.611346    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-d44c8cbb7-lmw97"
Oct 30 21:42:27.612609 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.612589    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-tlwms"
Oct 30 21:42:27.616057 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.616028    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-j5xsr"
Oct 30 21:42:27.617554 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.617518    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q6tjc"
Oct 30 21:42:27.622271 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.622253    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-b6cdbd45d-bdftd"
Oct 30 21:42:27.623244 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.623225    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6pttx"
Oct 30 21:42:27.625681 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.625667    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-mstdj"
Oct 30 21:42:27.656927 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.656898    5061 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 21:42:27.657022 ip-10-0-130-60 kubenswrapper[5061]: E1030 21:42:27.656956    5061 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-130-60.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-60.us-west-2.compute.internal"
Oct 30 21:42:27.659719 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.659696    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Oct 30 21:42:27.659821 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.659740    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Oct 30 21:42:27.660111 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.660091    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Oct 30 21:42:27.660380 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.660367    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Oct 30 21:42:27.660648 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.660635    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-5m5gq\""
Oct 30 21:42:27.661645 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.661624    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Oct 30 21:42:27.661738 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.661694    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Oct 30 21:42:27.661738 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.661717    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Oct 30 21:42:27.661841 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.661744    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-4rd6j\""
Oct 30 21:42:27.661841 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.661751    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Oct 30 21:42:27.661841 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.661756    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 30 21:42:27.661841 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.661768    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 21:42:27.661841 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.661756    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 21:42:27.661841 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.661785    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 21:42:27.661841 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.661784    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 21:42:27.661841 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.661806    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-rzsz9\""
Oct 30 21:42:27.661841 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.661845    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Oct 30 21:42:27.683743 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.683722    5061 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 21:42:27.708882 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.708856    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Oct 30 21:42:27.709335 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.709259    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-ngcxp\""
Oct 30 21:42:27.709469 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.709454    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-rqz2k\""
Oct 30 21:42:27.709864 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.709787    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Oct 30 21:42:27.709864 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.709827    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 21:42:27.709864 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.709846    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Oct 30 21:42:27.710017 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.709918    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Oct 30 21:42:27.710232 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.709457    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Oct 30 21:42:27.710367 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.710342    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 30 21:42:27.710415 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.709852    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 21:42:27.710415 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.710396    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 30 21:42:27.710720 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.710621    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Oct 30 21:42:27.710720 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.710672    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 30 21:42:27.710900 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.710773    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-g4k97\""
Oct 30 21:42:27.710996 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.710902    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Oct 30 21:42:27.710996 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.710935    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 30 21:42:27.711123 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.710997    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 30 21:42:27.711208 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.711191    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-zqhcb\""
Oct 30 21:42:27.711513 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.711484    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 21:42:27.711830 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.711810    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 21:42:27.711897 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.711870    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 21:42:27.711943 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.711925    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 21:42:27.712113 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.712057    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Oct 30 21:42:27.712301 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.712100    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-s2ds7\""
Oct 30 21:42:27.712559 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.712541    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 21:42:27.712559 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.712554    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 30 21:42:27.712690 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.712603    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Oct 30 21:42:27.712690 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.712633    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Oct 30 21:42:27.712690 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.712654    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 21:42:27.712840 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.712709    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Oct 30 21:42:27.712840 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.712711    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Oct 30 21:42:27.712840 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.712545    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 21:42:27.712966 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.712887    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 21:42:27.712966 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.712920    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Oct 30 21:42:27.713069 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.712981    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 21:42:27.713069 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.713002    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Oct 30 21:42:27.713138 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.713106    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-hmtv9\""
Oct 30 21:42:27.713138 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.713118    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-k8d6g\""
Oct 30 21:42:27.713201 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.713178    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-2qk7p\""
Oct 30 21:42:27.713201 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.713192    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Oct 30 21:42:27.713287 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.713228    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Oct 30 21:42:27.713287 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.713264    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 21:42:27.713357 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.713302    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-wvvdp\""
Oct 30 21:42:27.713398 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.713354    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 21:42:27.713447 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.713003    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-cnbh6\""
Oct 30 21:42:27.713447 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.713422    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-kphrd\""
Oct 30 21:42:27.713513 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.713497    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Oct 30 21:42:27.713912 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.713889    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 21:42:27.713982 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.713928    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Oct 30 21:42:27.713982 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.713941    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-58hb8\""
Oct 30 21:42:27.713982 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.713945    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Oct 30 21:42:27.713982 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.713968    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Oct 30 21:42:27.713982 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.713975    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 21:42:27.713982 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.713983    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 21:42:27.714294 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.713999    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Oct 30 21:42:27.714294 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.714009    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Oct 30 21:42:27.714294 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.714014    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 30 21:42:27.714294 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.714040    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-twk8z\""
Oct 30 21:42:27.714294 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.714092    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-ssgdc\""
Oct 30 21:42:27.714294 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.714105    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 21:42:27.714294 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.714182    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-48hj5\""
Oct 30 21:42:27.714294 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.714187    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 21:42:27.714294 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.714232    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 21:42:27.714294 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.714248    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6s2r8\""
Oct 30 21:42:27.714294 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.714299    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Oct 30 21:42:27.714781 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.714325    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 21:42:27.714781 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.714425    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-j86j7\""
Oct 30 21:42:27.714781 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.714460    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 21:42:27.714781 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.714469    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 30 21:42:27.714781 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.714483    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 21:42:27.714781 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.714492    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 21:42:27.714781 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.714513    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 21:42:27.714781 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.714607    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-mxjzl\""
Oct 30 21:42:27.714781 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.714656    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-xf9tm\""
Oct 30 21:42:27.714781 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.714696    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Oct 30 21:42:27.714781 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.714771    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 21:42:27.715237 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.714828    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 21:42:27.715237 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.714851    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-jxkz7\""
Oct 30 21:42:27.715237 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.714875    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 30 21:42:27.715237 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.714879    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 21:42:27.715237 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.714927    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 21:42:27.715237 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.714946    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 21:42:27.715237 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.714965    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-hdwjf\""
Oct 30 21:42:27.715237 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.715007    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 21:42:27.715237 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.715120    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 21:42:27.715237 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.715135    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 21:42:27.715237 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.715221    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 21:42:27.715237 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.715224    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 21:42:27.715588 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.715248    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 21:42:27.715588 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.715334    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-8c5b8\""
Oct 30 21:42:27.715588 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.715374    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-rx46t\""
Oct 30 21:42:27.715588 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.715395    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 21:42:27.715588 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.715422    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 21:42:27.715588 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.715489    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-vwlkq\""
Oct 30 21:42:27.715588 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.715549    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1f3b6a23-d7ea-4ce4-b1e8-68133c4b4ebc-system-cni-dir\") pod \"multus-additional-cni-plugins-b47v2\" (UID: \"1f3b6a23-d7ea-4ce4-b1e8-68133c4b4ebc\") " pod="openshift-multus/multus-additional-cni-plugins-b47v2"
Oct 30 21:42:27.715588 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.715557    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-5r6kl\""
Oct 30 21:42:27.715588 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.715567    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/538d5103-1c92-48f1-acc7-f90956848727-systemd-units\") pod \"ovnkube-node-22ldv\" (UID: \"538d5103-1c92-48f1-acc7-f90956848727\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ldv"
Oct 30 21:42:27.715973 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.715654    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cad67bc5-c8d2-497c-ba1b-cdbae5822cfa-image-registry-private-configuration\") pod \"image-registry-b6cdbd45d-bdftd\" (UID: \"cad67bc5-c8d2-497c-ba1b-cdbae5822cfa\") " pod="openshift-image-registry/image-registry-b6cdbd45d-bdftd"
Oct 30 21:42:27.715973 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.715690    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpbtv\" (UniqueName: \"kubernetes.io/projected/b227ad6a-9e59-415e-b7e6-f7c25e988abb-kube-api-access-qpbtv\") pod \"node-ca-x8gnj\" (UID: \"b227ad6a-9e59-415e-b7e6-f7c25e988abb\") " pod="openshift-image-registry/node-ca-x8gnj"
Oct 30 21:42:27.715973 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.715724    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Oct 30 21:42:27.715973 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.715719    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a2e3b25e-b6dc-46ef-86e8-4e70f9dad200-multus-socket-dir-parent\") pod \"multus-4nqkj\" (UID: \"a2e3b25e-b6dc-46ef-86e8-4e70f9dad200\") " pod="openshift-multus/multus-4nqkj"
Oct 30 21:42:27.715973 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.715755    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a2e3b25e-b6dc-46ef-86e8-4e70f9dad200-multus-daemon-config\") pod \"multus-4nqkj\" (UID: \"a2e3b25e-b6dc-46ef-86e8-4e70f9dad200\") " pod="openshift-multus/multus-4nqkj"
Oct 30 21:42:27.715973 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.715781    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/74ff988a-c05e-4a3e-adff-5945b79cf248-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mstdj\" (UID: \"74ff988a-c05e-4a3e-adff-5945b79cf248\") " pod="openshift-insights/insights-runtime-extractor-mstdj"
Oct 30 21:42:27.715973 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.715808    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/102b8049-7349-4e2b-974a-44c901eebcf3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-ns759\" (UID: \"102b8049-7349-4e2b-974a-44c901eebcf3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ns759"
Oct 30 21:42:27.715973 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.715855    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 21:42:27.715973 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.715856    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cad67bc5-c8d2-497c-ba1b-cdbae5822cfa-registry-certificates\") pod \"image-registry-b6cdbd45d-bdftd\" (UID: \"cad67bc5-c8d2-497c-ba1b-cdbae5822cfa\") " pod="openshift-image-registry/image-registry-b6cdbd45d-bdftd"
Oct 30 21:42:27.715973 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.715908    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cad67bc5-c8d2-497c-ba1b-cdbae5822cfa-image-registry-private-configuration\") pod \"image-registry-b6cdbd45d-bdftd\" (UID: \"cad67bc5-c8d2-497c-ba1b-cdbae5822cfa\") " pod="openshift-image-registry/image-registry-b6cdbd45d-bdftd"
Oct 30 21:42:27.715973 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.715918    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9grk\" (UniqueName: \"kubernetes.io/projected/033356b5-3c05-4f49-ad6a-7c369259970d-kube-api-access-d9grk\") pod \"migrator-66df5d496b-q4khr\" (UID: \"033356b5-3c05-4f49-ad6a-7c369259970d\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-q4khr"
Oct 30 21:42:27.715973 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.715958    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/613e7c8a-e2c7-424a-93e0-7fa86d18a8a9-registry-certificates\") pod \"image-registry-d44c8cbb7-lmw97\" (UID: \"613e7c8a-e2c7-424a-93e0-7fa86d18a8a9\") " pod="openshift-image-registry/image-registry-d44c8cbb7-lmw97"
Oct 30 21:42:27.716581 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.716039    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cad67bc5-c8d2-497c-ba1b-cdbae5822cfa-registry-certificates\") pod \"image-registry-b6cdbd45d-bdftd\" (UID: \"cad67bc5-c8d2-497c-ba1b-cdbae5822cfa\") " pod="openshift-image-registry/image-registry-b6cdbd45d-bdftd"
Oct 30 21:42:27.716581 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.716062    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-579v9\" (UniqueName: \"kubernetes.io/projected/3784e680-5efe-4ffb-ad71-956d05b57dce-kube-api-access-579v9\") pod \"network-check-target-979r9\" (UID: \"3784e680-5efe-4ffb-ad71-956d05b57dce\") " pod="openshift-network-diagnostics/network-check-target-979r9"
Oct 30 21:42:27.716581 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.716095    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d-default-certificate\") pod \"router-default-6c7f96b45-4wj99\" (UID: \"9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d\") " pod="openshift-ingress/router-default-6c7f96b45-4wj99"
Oct 30 21:42:27.716581 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.716190    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/613e7c8a-e2c7-424a-93e0-7fa86d18a8a9-registry-certificates\") pod \"image-registry-d44c8cbb7-lmw97\" (UID: \"613e7c8a-e2c7-424a-93e0-7fa86d18a8a9\") " pod="openshift-image-registry/image-registry-d44c8cbb7-lmw97"
Oct 30 21:42:27.716581 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.716201    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/613e7c8a-e2c7-424a-93e0-7fa86d18a8a9-trusted-ca\") pod \"image-registry-d44c8cbb7-lmw97\" (UID: \"613e7c8a-e2c7-424a-93e0-7fa86d18a8a9\") " pod="openshift-image-registry/image-registry-d44c8cbb7-lmw97"
Oct 30 21:42:27.716581 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.716233    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ed08bc51-70e6-4658-adb3-ed36f5176552-lib-modules\") pod \"tuned-hskxk\" (UID: \"ed08bc51-70e6-4658-adb3-ed36f5176552\") " pod="openshift-cluster-node-tuning-operator/tuned-hskxk"
Oct 30 21:42:27.716581 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.716240    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a2e3b25e-b6dc-46ef-86e8-4e70f9dad200-multus-daemon-config\") pod \"multus-4nqkj\" (UID: \"a2e3b25e-b6dc-46ef-86e8-4e70f9dad200\") " pod="openshift-multus/multus-4nqkj"
Oct 30 21:42:27.716581 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.716257    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ed08bc51-70e6-4658-adb3-ed36f5176552-etc-tuned\") pod \"tuned-hskxk\" (UID: \"ed08bc51-70e6-4658-adb3-ed36f5176552\") " pod="openshift-cluster-node-tuning-operator/tuned-hskxk"
Oct 30 21:42:27.716581 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.716280    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ed08bc51-70e6-4658-adb3-ed36f5176552-tmp\") pod \"tuned-hskxk\" (UID: \"ed08bc51-70e6-4658-adb3-ed36f5176552\") " pod="openshift-cluster-node-tuning-operator/tuned-hskxk"
Oct 30 21:42:27.716581 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.716315    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fe41c70b-7e2c-432f-8389-0282d6fdb332-original-pull-secret\") pod \"global-pull-secret-syncer-6wbw4\" (UID: \"fe41c70b-7e2c-432f-8389-0282d6fdb332\") " pod="kube-system/global-pull-secret-syncer-6wbw4"
Oct 30 21:42:27.716581 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.716358    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xm5l5\" (UniqueName: \"kubernetes.io/projected/101c697b-d7b8-4cfe-9d4c-818b5c04108f-kube-api-access-xm5l5\") pod \"periodic-gathering-7t4kc-b65p7\" (UID: \"101c697b-d7b8-4cfe-9d4c-818b5c04108f\") " pod="openshift-insights/periodic-gathering-7t4kc-b65p7"
Oct 30 21:42:27.716581 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.716376    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d-default-certificate\") pod \"router-default-6c7f96b45-4wj99\" (UID: \"9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d\") " pod="openshift-ingress/router-default-6c7f96b45-4wj99"
Oct 30 21:42:27.716581 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.716385    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/84d9b682-3241-40af-87d5-f87d54ce3697-config-volume\") pod \"dns-default-6pttx\" (UID: \"84d9b682-3241-40af-87d5-f87d54ce3697\") " pod="openshift-dns/dns-default-6pttx"
Oct 30 21:42:27.716581 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.716415    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ed08bc51-70e6-4658-adb3-ed36f5176552-etc-tuned\") pod \"tuned-hskxk\" (UID: \"ed08bc51-70e6-4658-adb3-ed36f5176552\") " pod="openshift-cluster-node-tuning-operator/tuned-hskxk"
Oct 30 21:42:27.716581 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.716417    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d-service-ca-bundle\") pod \"router-default-6c7f96b45-4wj99\" (UID: \"9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d\") " pod="openshift-ingress/router-default-6c7f96b45-4wj99"
Oct 30 21:42:27.716581 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.716450    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8w7g2\" (UniqueName: \"kubernetes.io/projected/798f248f-d429-4f56-8752-e157dd94206b-kube-api-access-8w7g2\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-mgxcj\" (UID: \"798f248f-d429-4f56-8752-e157dd94206b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-mgxcj"
Oct 30 21:42:27.716581 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.716497    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/613e7c8a-e2c7-424a-93e0-7fa86d18a8a9-trusted-ca\") pod \"image-registry-d44c8cbb7-lmw97\" (UID: \"613e7c8a-e2c7-424a-93e0-7fa86d18a8a9\") " pod="openshift-image-registry/image-registry-d44c8cbb7-lmw97"
Oct 30 21:42:27.716581 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.716515    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/613e7c8a-e2c7-424a-93e0-7fa86d18a8a9-ca-trust-extracted\") pod \"image-registry-d44c8cbb7-lmw97\" (UID: \"613e7c8a-e2c7-424a-93e0-7fa86d18a8a9\") " pod="openshift-image-registry/image-registry-d44c8cbb7-lmw97"
Oct 30 21:42:27.716581 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.716376    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ed08bc51-70e6-4658-adb3-ed36f5176552-tmp\") pod \"tuned-hskxk\" (UID: \"ed08bc51-70e6-4658-adb3-ed36f5176552\") " pod="openshift-cluster-node-tuning-operator/tuned-hskxk"
Oct 30 21:42:27.716581 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.716562    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ed08bc51-70e6-4658-adb3-ed36f5176552-etc-sysctl-conf\") pod \"tuned-hskxk\" (UID: \"ed08bc51-70e6-4658-adb3-ed36f5176552\") " pod="openshift-cluster-node-tuning-operator/tuned-hskxk"
Oct 30 21:42:27.716581 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.716574    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fe41c70b-7e2c-432f-8389-0282d6fdb332-original-pull-secret\") pod \"global-pull-secret-syncer-6wbw4\" (UID: \"fe41c70b-7e2c-432f-8389-0282d6fdb332\") " pod="kube-system/global-pull-secret-syncer-6wbw4"
Oct 30 21:42:27.716581 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.716583    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/84d9b682-3241-40af-87d5-f87d54ce3697-config-volume\") pod \"dns-default-6pttx\" (UID: \"84d9b682-3241-40af-87d5-f87d54ce3697\") " pod="openshift-dns/dns-default-6pttx"
Oct 30 21:42:27.716581 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.716596    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/71ddf455-cbf7-49a6-9773-30f7e4997a39-hosts-file\") pod \"node-resolver-2wsrr\" (UID: \"71ddf455-cbf7-49a6-9773-30f7e4997a39\") " pod="openshift-dns/node-resolver-2wsrr"
Oct 30 21:42:27.717436 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.716606    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/613e7c8a-e2c7-424a-93e0-7fa86d18a8a9-ca-trust-extracted\") pod \"image-registry-d44c8cbb7-lmw97\" (UID: \"613e7c8a-e2c7-424a-93e0-7fa86d18a8a9\") " pod="openshift-image-registry/image-registry-d44c8cbb7-lmw97"
Oct 30 21:42:27.717436 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.716698    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a2e3b25e-b6dc-46ef-86e8-4e70f9dad200-cnibin\") pod \"multus-4nqkj\" (UID: \"a2e3b25e-b6dc-46ef-86e8-4e70f9dad200\") " pod="openshift-multus/multus-4nqkj"
Oct 30 21:42:27.717436 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.716716    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4427f59d-8003-440f-9646-a46b176cc4a7-etc-selinux\") pod \"aws-ebs-csi-driver-node-qk9dg\" (UID: \"4427f59d-8003-440f-9646-a46b176cc4a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qk9dg"
Oct 30 21:42:27.717436 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.716748    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fe41c70b-7e2c-432f-8389-0282d6fdb332-kubelet-config\") pod \"global-pull-secret-syncer-6wbw4\" (UID: \"fe41c70b-7e2c-432f-8389-0282d6fdb332\") " pod="kube-system/global-pull-secret-syncer-6wbw4"
Oct 30 21:42:27.717436 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.716779    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/238ff7d8-7da8-4cd8-9d45-deedf615211e-konnectivity-ca\") pod \"konnectivity-agent-sztmx\" (UID: \"238ff7d8-7da8-4cd8-9d45-deedf615211e\") " pod="kube-system/konnectivity-agent-sztmx"
Oct 30 21:42:27.717436 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.716798    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/74ff988a-c05e-4a3e-adff-5945b79cf248-data-volume\") pod \"insights-runtime-extractor-mstdj\" (UID: \"74ff988a-c05e-4a3e-adff-5945b79cf248\") " pod="openshift-insights/insights-runtime-extractor-mstdj"
Oct 30 21:42:27.717436 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.716826    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhbrj\" (UniqueName: \"kubernetes.io/projected/d700d1eb-fbf4-45df-9b57-eca08d87ebe2-kube-api-access-nhbrj\") pod \"volume-data-source-validator-58c9f5f46d-s6427\" (UID: \"d700d1eb-fbf4-45df-9b57-eca08d87ebe2\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-s6427"
Oct 30 21:42:27.717436 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.716923    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/538d5103-1c92-48f1-acc7-f90956848727-host-slash\") pod \"ovnkube-node-22ldv\" (UID: \"538d5103-1c92-48f1-acc7-f90956848727\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ldv"
Oct 30 21:42:27.717436 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.716943    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/74ff988a-c05e-4a3e-adff-5945b79cf248-data-volume\") pod \"insights-runtime-extractor-mstdj\" (UID: \"74ff988a-c05e-4a3e-adff-5945b79cf248\") " pod="openshift-insights/insights-runtime-extractor-mstdj"
Oct 30 21:42:27.717436 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.716974    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/538d5103-1c92-48f1-acc7-f90956848727-host-run-ovn-kubernetes\") pod \"ovnkube-node-22ldv\" (UID: \"538d5103-1c92-48f1-acc7-f90956848727\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ldv"
Oct 30 21:42:27.717436 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.717019    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a2e3b25e-b6dc-46ef-86e8-4e70f9dad200-host-run-netns\") pod \"multus-4nqkj\" (UID: \"a2e3b25e-b6dc-46ef-86e8-4e70f9dad200\") " pod="openshift-multus/multus-4nqkj"
Oct 30 21:42:27.717436 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.717054    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-424gm\" (UniqueName: \"kubernetes.io/projected/a32aacdc-53be-41cf-839e-435d43361890-kube-api-access-424gm\") pod \"kas-connection-checker-685895c7c9-cklcw\" (UID: \"a32aacdc-53be-41cf-839e-435d43361890\") " pod="kube-system/kas-connection-checker-685895c7c9-cklcw"
Oct 30 21:42:27.717436 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.717087    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/238ff7d8-7da8-4cd8-9d45-deedf615211e-konnectivity-ca\") pod \"konnectivity-agent-sztmx\" (UID: \"238ff7d8-7da8-4cd8-9d45-deedf615211e\") " pod="kube-system/konnectivity-agent-sztmx"
Oct 30 21:42:27.717436 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.717095    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrpz5\" (UniqueName: \"kubernetes.io/projected/538d5103-1c92-48f1-acc7-f90956848727-kube-api-access-hrpz5\") pod \"ovnkube-node-22ldv\" (UID: \"538d5103-1c92-48f1-acc7-f90956848727\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ldv"
Oct 30 21:42:27.717436 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.717186    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/c2386477-903e-4c6c-b807-3f473a58a6b6-signing-key\") pod \"service-ca-5fffc97bdf-th574\" (UID: \"c2386477-903e-4c6c-b807-3f473a58a6b6\") " pod="openshift-service-ca/service-ca-5fffc97bdf-th574"
Oct 30 21:42:27.717436 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.717202    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mms6q\" (UniqueName: \"kubernetes.io/projected/84d9b682-3241-40af-87d5-f87d54ce3697-kube-api-access-mms6q\") pod \"dns-default-6pttx\" (UID: \"84d9b682-3241-40af-87d5-f87d54ce3697\") " pod="openshift-dns/dns-default-6pttx"
Oct 30 21:42:27.717436 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.717248    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4658n\" (UniqueName: \"kubernetes.io/projected/879022d4-06b9-44cc-a2a2-dc307b27345e-kube-api-access-4658n\") pod \"migrator-66df5d496b-rznzd\" (UID: \"879022d4-06b9-44cc-a2a2-dc307b27345e\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-rznzd"
Oct 30 21:42:27.717436 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.717353    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d-stats-auth\") pod \"router-default-6c7f96b45-4wj99\" (UID: \"9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d\") " pod="openshift-ingress/router-default-6c7f96b45-4wj99"
Oct 30 21:42:27.717436 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.717403    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5e74071e-0462-42b5-a05c-d90bb638478a-config\") pod \"console-operator-7cd56b8dbc-sclrt\" (UID: \"5e74071e-0462-42b5-a05c-d90bb638478a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-sclrt"
Oct 30 21:42:27.717436 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.717427    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/c2386477-903e-4c6c-b807-3f473a58a6b6-signing-key\") pod \"service-ca-5fffc97bdf-th574\" (UID: \"c2386477-903e-4c6c-b807-3f473a58a6b6\") " pod="openshift-service-ca/service-ca-5fffc97bdf-th574"
Oct 30 21:42:27.718181 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.717458    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a2e3b25e-b6dc-46ef-86e8-4e70f9dad200-system-cni-dir\") pod \"multus-4nqkj\" (UID: \"a2e3b25e-b6dc-46ef-86e8-4e70f9dad200\") " pod="openshift-multus/multus-4nqkj"
Oct 30 21:42:27.718181 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.717568    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a2e3b25e-b6dc-46ef-86e8-4e70f9dad200-etc-kubernetes\") pod \"multus-4nqkj\" (UID: \"a2e3b25e-b6dc-46ef-86e8-4e70f9dad200\") " pod="openshift-multus/multus-4nqkj"
Oct 30 21:42:27.718181 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.717598    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d-stats-auth\") pod \"router-default-6c7f96b45-4wj99\" (UID: \"9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d\") " pod="openshift-ingress/router-default-6c7f96b45-4wj99"
Oct 30 21:42:27.718181 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.717607    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0929b9f6-12ec-4f4d-8ba3-5cef9defba15-iptables-alerter-script\") pod \"iptables-alerter-wbq4l\" (UID: \"0929b9f6-12ec-4f4d-8ba3-5cef9defba15\") " pod="openshift-network-operator/iptables-alerter-wbq4l"
Oct 30 21:42:27.718181 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.717653    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5e74071e-0462-42b5-a05c-d90bb638478a-config\") pod \"console-operator-7cd56b8dbc-sclrt\" (UID: \"5e74071e-0462-42b5-a05c-d90bb638478a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-sclrt"
Oct 30 21:42:27.718181 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.717667    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fcz7\" (UniqueName: \"kubernetes.io/projected/102b8049-7349-4e2b-974a-44c901eebcf3-kube-api-access-5fcz7\") pod \"cluster-monitoring-operator-884576c57-ns759\" (UID: \"102b8049-7349-4e2b-974a-44c901eebcf3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ns759"
Oct 30 21:42:27.718181 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.717742    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbwzr\" (UniqueName: \"kubernetes.io/projected/2205e8e6-f038-47fa-b393-0abe4d622e03-kube-api-access-rbwzr\") pod \"network-metrics-daemon-q6tjc\" (UID: \"2205e8e6-f038-47fa-b393-0abe4d622e03\") " pod="openshift-multus/network-metrics-daemon-q6tjc"
Oct 30 21:42:27.718181 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.717856    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/538d5103-1c92-48f1-acc7-f90956848727-run-openvswitch\") pod \"ovnkube-node-22ldv\" (UID: \"538d5103-1c92-48f1-acc7-f90956848727\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ldv"
Oct 30 21:42:27.718181 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.717867    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0929b9f6-12ec-4f4d-8ba3-5cef9defba15-iptables-alerter-script\") pod \"iptables-alerter-wbq4l\" (UID: \"0929b9f6-12ec-4f4d-8ba3-5cef9defba15\") " pod="openshift-network-operator/iptables-alerter-wbq4l"
Oct 30 21:42:27.718181 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.717883    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ed08bc51-70e6-4658-adb3-ed36f5176552-etc-modprobe-d\") pod \"tuned-hskxk\" (UID: \"ed08bc51-70e6-4658-adb3-ed36f5176552\") " pod="openshift-cluster-node-tuning-operator/tuned-hskxk"
Oct 30 21:42:27.718181 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.717922    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5e74071e-0462-42b5-a05c-d90bb638478a-serving-cert\") pod \"console-operator-7cd56b8dbc-sclrt\" (UID: \"5e74071e-0462-42b5-a05c-d90bb638478a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-sclrt"
Oct 30 21:42:27.718181 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.717962    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b227ad6a-9e59-415e-b7e6-f7c25e988abb-host\") pod \"node-ca-x8gnj\" (UID: \"b227ad6a-9e59-415e-b7e6-f7c25e988abb\") " pod="openshift-image-registry/node-ca-x8gnj"
Oct 30 21:42:27.718181 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.717987    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srwt7\" (UniqueName: \"kubernetes.io/projected/0126a0a0-a1c8-4db3-a6af-629360e247f4-kube-api-access-srwt7\") pod \"service-ca-operator-544d749c8c-vqwlt\" (UID: \"0126a0a0-a1c8-4db3-a6af-629360e247f4\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vqwlt"
Oct 30 21:42:27.718181 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.718094    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/798f248f-d429-4f56-8752-e157dd94206b-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-mgxcj\" (UID: \"798f248f-d429-4f56-8752-e157dd94206b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-mgxcj"
Oct 30 21:42:27.718181 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.718127    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/798f248f-d429-4f56-8752-e157dd94206b-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-mgxcj\" (UID: \"798f248f-d429-4f56-8752-e157dd94206b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-mgxcj"
Oct 30 21:42:27.718181 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.718153    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/538d5103-1c92-48f1-acc7-f90956848727-env-overrides\") pod \"ovnkube-node-22ldv\" (UID: \"538d5103-1c92-48f1-acc7-f90956848727\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ldv"
Oct 30 21:42:27.718181 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.718178    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/238ff7d8-7da8-4cd8-9d45-deedf615211e-agent-certs\") pod \"konnectivity-agent-sztmx\" (UID: \"238ff7d8-7da8-4cd8-9d45-deedf615211e\") " pod="kube-system/konnectivity-agent-sztmx"
Oct 30 21:42:27.718723 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.718204    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swjcw\" (UniqueName: \"kubernetes.io/projected/0929b9f6-12ec-4f4d-8ba3-5cef9defba15-kube-api-access-swjcw\") pod \"iptables-alerter-wbq4l\" (UID: \"0929b9f6-12ec-4f4d-8ba3-5cef9defba15\") " pod="openshift-network-operator/iptables-alerter-wbq4l"
Oct 30 21:42:27.718723 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.718230    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5e74071e-0462-42b5-a05c-d90bb638478a-trusted-ca\") pod \"console-operator-7cd56b8dbc-sclrt\" (UID: \"5e74071e-0462-42b5-a05c-d90bb638478a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-sclrt"
Oct 30 21:42:27.718723 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.718247    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5e74071e-0462-42b5-a05c-d90bb638478a-serving-cert\") pod \"console-operator-7cd56b8dbc-sclrt\" (UID: \"5e74071e-0462-42b5-a05c-d90bb638478a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-sclrt"
Oct 30 21:42:27.718723 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.718257    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4427f59d-8003-440f-9646-a46b176cc4a7-device-dir\") pod \"aws-ebs-csi-driver-node-qk9dg\" (UID: \"4427f59d-8003-440f-9646-a46b176cc4a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qk9dg"
Oct 30 21:42:27.718723 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.718296    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qd5b8\" (UniqueName: \"kubernetes.io/projected/c69312da-8be3-4931-8a81-a8ff3f86d5e6-kube-api-access-qd5b8\") pod \"insights-operator-7c4986d449-l9b9d\" (UID: \"c69312da-8be3-4931-8a81-a8ff3f86d5e6\") " pod="openshift-insights/insights-operator-7c4986d449-l9b9d"
Oct 30 21:42:27.718723 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.718332    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/798f248f-d429-4f56-8752-e157dd94206b-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-mgxcj\" (UID: \"798f248f-d429-4f56-8752-e157dd94206b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-mgxcj"
Oct 30 21:42:27.718723 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.718369    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4tb7\" (UniqueName: \"kubernetes.io/projected/9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d-kube-api-access-n4tb7\") pod \"router-default-6c7f96b45-4wj99\" (UID: \"9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d\") " pod="openshift-ingress/router-default-6c7f96b45-4wj99"
Oct 30 21:42:27.718723 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.718397    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/538d5103-1c92-48f1-acc7-f90956848727-host-cni-netd\") pod \"ovnkube-node-22ldv\" (UID: \"538d5103-1c92-48f1-acc7-f90956848727\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ldv"
Oct 30 21:42:27.718723 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.718421    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/538d5103-1c92-48f1-acc7-f90956848727-env-overrides\") pod \"ovnkube-node-22ldv\" (UID: \"538d5103-1c92-48f1-acc7-f90956848727\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ldv"
Oct 30 21:42:27.718723 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.718437    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cad67bc5-c8d2-497c-ba1b-cdbae5822cfa-bound-sa-token\") pod \"image-registry-b6cdbd45d-bdftd\" (UID: \"cad67bc5-c8d2-497c-ba1b-cdbae5822cfa\") " pod="openshift-image-registry/image-registry-b6cdbd45d-bdftd"
Oct 30 21:42:27.718723 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.718456    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/798f248f-d429-4f56-8752-e157dd94206b-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-mgxcj\" (UID: \"798f248f-d429-4f56-8752-e157dd94206b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-mgxcj"
Oct 30 21:42:27.718723 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.718462    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fe41c70b-7e2c-432f-8389-0282d6fdb332-dbus\") pod \"global-pull-secret-syncer-6wbw4\" (UID: \"fe41c70b-7e2c-432f-8389-0282d6fdb332\") " pod="kube-system/global-pull-secret-syncer-6wbw4"
Oct 30 21:42:27.718723 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.718519    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/238ff7d8-7da8-4cd8-9d45-deedf615211e-agent-certs\") pod \"konnectivity-agent-sztmx\" (UID: \"238ff7d8-7da8-4cd8-9d45-deedf615211e\") " pod="kube-system/konnectivity-agent-sztmx"
Oct 30 21:42:27.718723 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.718603    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdbmm\" (UniqueName: \"kubernetes.io/projected/0bb51175-fa5e-4472-8d83-106bca721585-kube-api-access-pdbmm\") pod \"cluster-samples-operator-6ddb6c8b94-tlwms\" (UID: \"0bb51175-fa5e-4472-8d83-106bca721585\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-tlwms"
Oct 30 21:42:27.718723 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.718642    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a2e3b25e-b6dc-46ef-86e8-4e70f9dad200-host-var-lib-cni-multus\") pod \"multus-4nqkj\" (UID: \"a2e3b25e-b6dc-46ef-86e8-4e70f9dad200\") " pod="openshift-multus/multus-4nqkj"
Oct 30 21:42:27.718723 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.718690    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a2e3b25e-b6dc-46ef-86e8-4e70f9dad200-multus-conf-dir\") pod \"multus-4nqkj\" (UID: \"a2e3b25e-b6dc-46ef-86e8-4e70f9dad200\") " pod="openshift-multus/multus-4nqkj"
Oct 30 21:42:27.719247 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.718759    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/613e7c8a-e2c7-424a-93e0-7fa86d18a8a9-bound-sa-token\") pod \"image-registry-d44c8cbb7-lmw97\" (UID: \"613e7c8a-e2c7-424a-93e0-7fa86d18a8a9\") " pod="openshift-image-registry/image-registry-d44c8cbb7-lmw97"
Oct 30 21:42:27.719247 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.718920    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/538d5103-1c92-48f1-acc7-f90956848727-run-ovn\") pod \"ovnkube-node-22ldv\" (UID: \"538d5103-1c92-48f1-acc7-f90956848727\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ldv"
Oct 30 21:42:27.719247 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.718977    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/538d5103-1c92-48f1-acc7-f90956848727-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-22ldv\" (UID: \"538d5103-1c92-48f1-acc7-f90956848727\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ldv"
Oct 30 21:42:27.719247 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.719011    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zv2qw\" (UniqueName: \"kubernetes.io/projected/c2386477-903e-4c6c-b807-3f473a58a6b6-kube-api-access-zv2qw\") pod \"service-ca-5fffc97bdf-th574\" (UID: \"c2386477-903e-4c6c-b807-3f473a58a6b6\") " pod="openshift-service-ca/service-ca-5fffc97bdf-th574"
Oct 30 21:42:27.719247 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.719060    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5e74071e-0462-42b5-a05c-d90bb638478a-trusted-ca\") pod \"console-operator-7cd56b8dbc-sclrt\" (UID: \"5e74071e-0462-42b5-a05c-d90bb638478a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-sclrt"
Oct 30 21:42:27.719247 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.719076    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a2e3b25e-b6dc-46ef-86e8-4e70f9dad200-host-var-lib-kubelet\") pod \"multus-4nqkj\" (UID: \"a2e3b25e-b6dc-46ef-86e8-4e70f9dad200\") " pod="openshift-multus/multus-4nqkj"
Oct 30 21:42:27.719247 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.719114    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hkfh\" (UniqueName: \"kubernetes.io/projected/71ddf455-cbf7-49a6-9773-30f7e4997a39-kube-api-access-7hkfh\") pod \"node-resolver-2wsrr\" (UID: \"71ddf455-cbf7-49a6-9773-30f7e4997a39\") " pod="openshift-dns/node-resolver-2wsrr"
Oct 30 21:42:27.719247 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.719139    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/28210ad9-18f0-4ecf-9bfc-b256440f0ce8-cert\") pod \"ingress-canary-pzpgw\" (UID: \"28210ad9-18f0-4ecf-9bfc-b256440f0ce8\") " pod="openshift-ingress-canary/ingress-canary-pzpgw"
Oct 30 21:42:27.719582 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.719261    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a2e3b25e-b6dc-46ef-86e8-4e70f9dad200-hostroot\") pod \"multus-4nqkj\" (UID: \"a2e3b25e-b6dc-46ef-86e8-4e70f9dad200\") " pod="openshift-multus/multus-4nqkj"
Oct 30 21:42:27.719582 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.719293    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8b7v\" (UniqueName: \"kubernetes.io/projected/a2e3b25e-b6dc-46ef-86e8-4e70f9dad200-kube-api-access-p8b7v\") pod \"multus-4nqkj\" (UID: \"a2e3b25e-b6dc-46ef-86e8-4e70f9dad200\") " pod="openshift-multus/multus-4nqkj"
Oct 30 21:42:27.719582 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.719331    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b227ad6a-9e59-415e-b7e6-f7c25e988abb-serviceca\") pod \"node-ca-x8gnj\" (UID: \"b227ad6a-9e59-415e-b7e6-f7c25e988abb\") " pod="openshift-image-registry/node-ca-x8gnj"
Oct 30 21:42:27.719582 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.719343    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/28210ad9-18f0-4ecf-9bfc-b256440f0ce8-cert\") pod \"ingress-canary-pzpgw\" (UID: \"28210ad9-18f0-4ecf-9bfc-b256440f0ce8\") " pod="openshift-ingress-canary/ingress-canary-pzpgw"
Oct 30 21:42:27.719582 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.719357    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71ddf455-cbf7-49a6-9773-30f7e4997a39-tmp-dir\") pod \"node-resolver-2wsrr\" (UID: \"71ddf455-cbf7-49a6-9773-30f7e4997a39\") " pod="openshift-dns/node-resolver-2wsrr"
Oct 30 21:42:27.719582 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.719383    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4427f59d-8003-440f-9646-a46b176cc4a7-socket-dir\") pod \"aws-ebs-csi-driver-node-qk9dg\" (UID: \"4427f59d-8003-440f-9646-a46b176cc4a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qk9dg"
Oct 30 21:42:27.719582 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.719407    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1f3b6a23-d7ea-4ce4-b1e8-68133c4b4ebc-cnibin\") pod \"multus-additional-cni-plugins-b47v2\" (UID: \"1f3b6a23-d7ea-4ce4-b1e8-68133c4b4ebc\") " pod="openshift-multus/multus-additional-cni-plugins-b47v2"
Oct 30 21:42:27.719582 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.719434    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/c69312da-8be3-4931-8a81-a8ff3f86d5e6-snapshots\") pod \"insights-operator-7c4986d449-l9b9d\" (UID: \"c69312da-8be3-4931-8a81-a8ff3f86d5e6\") " pod="openshift-insights/insights-operator-7c4986d449-l9b9d"
Oct 30 21:42:27.719582 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.719474    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cad67bc5-c8d2-497c-ba1b-cdbae5822cfa-registry-tls\") pod \"image-registry-b6cdbd45d-bdftd\" (UID: \"cad67bc5-c8d2-497c-ba1b-cdbae5822cfa\") " pod="openshift-image-registry/image-registry-b6cdbd45d-bdftd"
Oct 30 21:42:27.719582 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.719482    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b227ad6a-9e59-415e-b7e6-f7c25e988abb-serviceca\") pod \"node-ca-x8gnj\" (UID: \"b227ad6a-9e59-415e-b7e6-f7c25e988abb\") " pod="openshift-image-registry/node-ca-x8gnj"
Oct 30 21:42:27.719582 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.719564    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71ddf455-cbf7-49a6-9773-30f7e4997a39-tmp-dir\") pod \"node-resolver-2wsrr\" (UID: \"71ddf455-cbf7-49a6-9773-30f7e4997a39\") " pod="openshift-dns/node-resolver-2wsrr"
Oct 30 21:42:27.719994 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.719643    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67qtb\" (UniqueName: \"kubernetes.io/projected/74ff988a-c05e-4a3e-adff-5945b79cf248-kube-api-access-67qtb\") pod \"insights-runtime-extractor-mstdj\" (UID: \"74ff988a-c05e-4a3e-adff-5945b79cf248\") " pod="openshift-insights/insights-runtime-extractor-mstdj"
Oct 30 21:42:27.719994 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.719668    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/c69312da-8be3-4931-8a81-a8ff3f86d5e6-snapshots\") pod \"insights-operator-7c4986d449-l9b9d\" (UID: \"c69312da-8be3-4931-8a81-a8ff3f86d5e6\") " pod="openshift-insights/insights-operator-7c4986d449-l9b9d"
Oct 30 21:42:27.719994 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.719684    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1f3b6a23-d7ea-4ce4-b1e8-68133c4b4ebc-cni-binary-copy\") pod \"multus-additional-cni-plugins-b47v2\" (UID: \"1f3b6a23-d7ea-4ce4-b1e8-68133c4b4ebc\") " pod="openshift-multus/multus-additional-cni-plugins-b47v2"
Oct 30 21:42:27.719994 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.719840    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c69312da-8be3-4931-8a81-a8ff3f86d5e6-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-l9b9d\" (UID: \"c69312da-8be3-4931-8a81-a8ff3f86d5e6\") " pod="openshift-insights/insights-operator-7c4986d449-l9b9d"
Oct 30 21:42:27.719994 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.719871    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f648e933-f035-486a-88d5-1b185c89782e-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-j5xsr\" (UID: \"f648e933-f035-486a-88d5-1b185c89782e\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-j5xsr"
Oct 30 21:42:27.719994 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.719899    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hw226\" (UniqueName: \"kubernetes.io/projected/613e7c8a-e2c7-424a-93e0-7fa86d18a8a9-kube-api-access-hw226\") pod \"image-registry-d44c8cbb7-lmw97\" (UID: \"613e7c8a-e2c7-424a-93e0-7fa86d18a8a9\") " pod="openshift-image-registry/image-registry-d44c8cbb7-lmw97"
Oct 30 21:42:27.719994 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.719930    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1f3b6a23-d7ea-4ce4-b1e8-68133c4b4ebc-cni-binary-copy\") pod \"multus-additional-cni-plugins-b47v2\" (UID: \"1f3b6a23-d7ea-4ce4-b1e8-68133c4b4ebc\") " pod="openshift-multus/multus-additional-cni-plugins-b47v2"
Oct 30 21:42:27.720289 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.720008    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ed08bc51-70e6-4658-adb3-ed36f5176552-etc-systemd\") pod \"tuned-hskxk\" (UID: \"ed08bc51-70e6-4658-adb3-ed36f5176552\") " pod="openshift-cluster-node-tuning-operator/tuned-hskxk"
Oct 30 21:42:27.720289 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.720051    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/101c697b-d7b8-4cfe-9d4c-818b5c04108f-service-ca-bundle\") pod \"periodic-gathering-7t4kc-b65p7\" (UID: \"101c697b-d7b8-4cfe-9d4c-818b5c04108f\") " pod="openshift-insights/periodic-gathering-7t4kc-b65p7"
Oct 30 21:42:27.720289 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.720088    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/101c697b-d7b8-4cfe-9d4c-818b5c04108f-trusted-ca-bundle\") pod \"periodic-gathering-7t4kc-b65p7\" (UID: \"101c697b-d7b8-4cfe-9d4c-818b5c04108f\") " pod="openshift-insights/periodic-gathering-7t4kc-b65p7"
Oct 30 21:42:27.720289 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.720170    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4427f59d-8003-440f-9646-a46b176cc4a7-registration-dir\") pod \"aws-ebs-csi-driver-node-qk9dg\" (UID: \"4427f59d-8003-440f-9646-a46b176cc4a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qk9dg"
Oct 30 21:42:27.720289 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.720209    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1f3b6a23-d7ea-4ce4-b1e8-68133c4b4ebc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-b47v2\" (UID: \"1f3b6a23-d7ea-4ce4-b1e8-68133c4b4ebc\") " pod="openshift-multus/multus-additional-cni-plugins-b47v2"
Oct 30 21:42:27.720289 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.720267    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdtc5\" (UniqueName: \"kubernetes.io/projected/3f97ee34-5dfc-454e-ba5c-afd939f3bd25-kube-api-access-cdtc5\") pod \"kas-connection-checker-685895c7c9-bgnkz\" (UID: \"3f97ee34-5dfc-454e-ba5c-afd939f3bd25\") " pod="kube-system/kas-connection-checker-685895c7c9-bgnkz"
Oct 30 21:42:27.720594 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.720309    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ed08bc51-70e6-4658-adb3-ed36f5176552-etc-sysctl-d\") pod \"tuned-hskxk\" (UID: \"ed08bc51-70e6-4658-adb3-ed36f5176552\") " pod="openshift-cluster-node-tuning-operator/tuned-hskxk"
Oct 30 21:42:27.720594 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.720323    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/101c697b-d7b8-4cfe-9d4c-818b5c04108f-service-ca-bundle\") pod \"periodic-gathering-7t4kc-b65p7\" (UID: \"101c697b-d7b8-4cfe-9d4c-818b5c04108f\") " pod="openshift-insights/periodic-gathering-7t4kc-b65p7"
Oct 30 21:42:27.720594 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.720341    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c69312da-8be3-4931-8a81-a8ff3f86d5e6-tmp\") pod \"insights-operator-7c4986d449-l9b9d\" (UID: \"c69312da-8be3-4931-8a81-a8ff3f86d5e6\") " pod="openshift-insights/insights-operator-7c4986d449-l9b9d"
Oct 30 21:42:27.720594 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.720366    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/538d5103-1c92-48f1-acc7-f90956848727-run-systemd\") pod \"ovnkube-node-22ldv\" (UID: \"538d5103-1c92-48f1-acc7-f90956848727\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ldv"
Oct 30 21:42:27.720594 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.720416    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9dpv\" (UniqueName: \"kubernetes.io/projected/cad67bc5-c8d2-497c-ba1b-cdbae5822cfa-kube-api-access-m9dpv\") pod \"image-registry-b6cdbd45d-bdftd\" (UID: \"cad67bc5-c8d2-497c-ba1b-cdbae5822cfa\") " pod="openshift-image-registry/image-registry-b6cdbd45d-bdftd"
Oct 30 21:42:27.720594 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.720481    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1f3b6a23-d7ea-4ce4-b1e8-68133c4b4ebc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-b47v2\" (UID: \"1f3b6a23-d7ea-4ce4-b1e8-68133c4b4ebc\") " pod="openshift-multus/multus-additional-cni-plugins-b47v2"
Oct 30 21:42:27.720594 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.720482    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/613e7c8a-e2c7-424a-93e0-7fa86d18a8a9-image-registry-private-configuration\") pod \"image-registry-d44c8cbb7-lmw97\" (UID: \"613e7c8a-e2c7-424a-93e0-7fa86d18a8a9\") " pod="openshift-image-registry/image-registry-d44c8cbb7-lmw97"
Oct 30 21:42:27.720594 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.720501    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c69312da-8be3-4931-8a81-a8ff3f86d5e6-tmp\") pod \"insights-operator-7c4986d449-l9b9d\" (UID: \"c69312da-8be3-4931-8a81-a8ff3f86d5e6\") " pod="openshift-insights/insights-operator-7c4986d449-l9b9d"
Oct 30 21:42:27.720594 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.720583    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ed08bc51-70e6-4658-adb3-ed36f5176552-sys\") pod \"tuned-hskxk\" (UID: \"ed08bc51-70e6-4658-adb3-ed36f5176552\") " pod="openshift-cluster-node-tuning-operator/tuned-hskxk"
Oct 30 21:42:27.721011 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.720620    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqmr4\" (UniqueName: \"kubernetes.io/projected/5e74071e-0462-42b5-a05c-d90bb638478a-kube-api-access-vqmr4\") pod \"console-operator-7cd56b8dbc-sclrt\" (UID: \"5e74071e-0462-42b5-a05c-d90bb638478a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-sclrt"
Oct 30 21:42:27.721011 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.720755    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c69312da-8be3-4931-8a81-a8ff3f86d5e6-service-ca-bundle\") pod \"insights-operator-7c4986d449-l9b9d\" (UID: \"c69312da-8be3-4931-8a81-a8ff3f86d5e6\") " pod="openshift-insights/insights-operator-7c4986d449-l9b9d"
Oct 30 21:42:27.721011 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.720765    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/613e7c8a-e2c7-424a-93e0-7fa86d18a8a9-image-registry-private-configuration\") pod \"image-registry-d44c8cbb7-lmw97\" (UID: \"613e7c8a-e2c7-424a-93e0-7fa86d18a8a9\") " pod="openshift-image-registry/image-registry-d44c8cbb7-lmw97"
Oct 30 21:42:27.721011 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.720889    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d-metrics-certs\") pod \"router-default-6c7f96b45-4wj99\" (UID: \"9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d\") " pod="openshift-ingress/router-default-6c7f96b45-4wj99"
Oct 30 21:42:27.721011 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.720923    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ed08bc51-70e6-4658-adb3-ed36f5176552-etc-kubernetes\") pod \"tuned-hskxk\" (UID: \"ed08bc51-70e6-4658-adb3-ed36f5176552\") " pod="openshift-cluster-node-tuning-operator/tuned-hskxk"
Oct 30 21:42:27.721011 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.720953    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/101c697b-d7b8-4cfe-9d4c-818b5c04108f-archives-path\") pod \"periodic-gathering-7t4kc-b65p7\" (UID: \"101c697b-d7b8-4cfe-9d4c-818b5c04108f\") " pod="openshift-insights/periodic-gathering-7t4kc-b65p7"
Oct 30 21:42:27.721011 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.720962    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c69312da-8be3-4931-8a81-a8ff3f86d5e6-service-ca-bundle\") pod \"insights-operator-7c4986d449-l9b9d\" (UID: \"c69312da-8be3-4931-8a81-a8ff3f86d5e6\") " pod="openshift-insights/insights-operator-7c4986d449-l9b9d"
Oct 30 21:42:27.721011 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.720998    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84d9b682-3241-40af-87d5-f87d54ce3697-tmp-dir\") pod \"dns-default-6pttx\" (UID: \"84d9b682-3241-40af-87d5-f87d54ce3697\") " pod="openshift-dns/dns-default-6pttx"
Oct 30 21:42:27.721339 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.721031    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0929b9f6-12ec-4f4d-8ba3-5cef9defba15-host-slash\") pod \"iptables-alerter-wbq4l\" (UID: \"0929b9f6-12ec-4f4d-8ba3-5cef9defba15\") " pod="openshift-network-operator/iptables-alerter-wbq4l"
Oct 30 21:42:27.721339 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.721040    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/101c697b-d7b8-4cfe-9d4c-818b5c04108f-archives-path\") pod \"periodic-gathering-7t4kc-b65p7\" (UID: \"101c697b-d7b8-4cfe-9d4c-818b5c04108f\") " pod="openshift-insights/periodic-gathering-7t4kc-b65p7"
Oct 30 21:42:27.721339 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.721055    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/0bb51175-fa5e-4472-8d83-106bca721585-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-tlwms\" (UID: \"0bb51175-fa5e-4472-8d83-106bca721585\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-tlwms"
Oct 30 21:42:27.721339 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.721078    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84d9b682-3241-40af-87d5-f87d54ce3697-tmp-dir\") pod \"dns-default-6pttx\" (UID: \"84d9b682-3241-40af-87d5-f87d54ce3697\") " pod="openshift-dns/dns-default-6pttx"
Oct 30 21:42:27.721339 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.721117    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/84d9b682-3241-40af-87d5-f87d54ce3697-metrics-tls\") pod \"dns-default-6pttx\" (UID: \"84d9b682-3241-40af-87d5-f87d54ce3697\") " pod="openshift-dns/dns-default-6pttx"
Oct 30 21:42:27.721339 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.721193    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/74ff988a-c05e-4a3e-adff-5945b79cf248-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mstdj\" (UID: \"74ff988a-c05e-4a3e-adff-5945b79cf248\") " pod="openshift-insights/insights-runtime-extractor-mstdj"
Oct 30 21:42:27.721339 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.721242    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c69312da-8be3-4931-8a81-a8ff3f86d5e6-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-l9b9d\" (UID: \"c69312da-8be3-4931-8a81-a8ff3f86d5e6\") " pod="openshift-insights/insights-operator-7c4986d449-l9b9d"
Oct 30 21:42:27.721339 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.721248    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0bb51175-fa5e-4472-8d83-106bca721585-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-tlwms\" (UID: \"0bb51175-fa5e-4472-8d83-106bca721585\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-tlwms"
Oct 30 21:42:27.721339 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.721296    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gq9m\" (UniqueName: \"kubernetes.io/projected/1f3b6a23-d7ea-4ce4-b1e8-68133c4b4ebc-kube-api-access-7gq9m\") pod \"multus-additional-cni-plugins-b47v2\" (UID: \"1f3b6a23-d7ea-4ce4-b1e8-68133c4b4ebc\") " pod="openshift-multus/multus-additional-cni-plugins-b47v2"
Oct 30 21:42:27.721339 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.721303    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/0bb51175-fa5e-4472-8d83-106bca721585-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-tlwms\" (UID: \"0bb51175-fa5e-4472-8d83-106bca721585\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-tlwms"
Oct 30 21:42:27.721729 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.721352    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/102b8049-7349-4e2b-974a-44c901eebcf3-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-ns759\" (UID: \"102b8049-7349-4e2b-974a-44c901eebcf3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ns759"
Oct 30 21:42:27.721729 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.721429    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/538d5103-1c92-48f1-acc7-f90956848727-host-kubelet\") pod \"ovnkube-node-22ldv\" (UID: \"538d5103-1c92-48f1-acc7-f90956848727\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ldv"
Oct 30 21:42:27.721729 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.721504    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/101c697b-d7b8-4cfe-9d4c-818b5c04108f-trusted-ca-bundle\") pod \"periodic-gathering-7t4kc-b65p7\" (UID: \"101c697b-d7b8-4cfe-9d4c-818b5c04108f\") " pod="openshift-insights/periodic-gathering-7t4kc-b65p7"
Oct 30 21:42:27.721729 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.721509    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/538d5103-1c92-48f1-acc7-f90956848727-host-run-netns\") pod \"ovnkube-node-22ldv\" (UID: \"538d5103-1c92-48f1-acc7-f90956848727\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ldv"
Oct 30 21:42:27.721729 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.721572    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/538d5103-1c92-48f1-acc7-f90956848727-ovn-node-metrics-cert\") pod \"ovnkube-node-22ldv\" (UID: \"538d5103-1c92-48f1-acc7-f90956848727\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ldv"
Oct 30 21:42:27.721729 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.721618    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4427f59d-8003-440f-9646-a46b176cc4a7-sys-fs\") pod \"aws-ebs-csi-driver-node-qk9dg\" (UID: \"4427f59d-8003-440f-9646-a46b176cc4a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qk9dg"
Oct 30 21:42:27.721729 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.721653    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/102b8049-7349-4e2b-974a-44c901eebcf3-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-ns759\" (UID: \"102b8049-7349-4e2b-974a-44c901eebcf3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ns759"
Oct 30 21:42:27.721729 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.721668    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/538d5103-1c92-48f1-acc7-f90956848727-etc-openvswitch\") pod \"ovnkube-node-22ldv\" (UID: \"538d5103-1c92-48f1-acc7-f90956848727\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ldv"
Oct 30 21:42:27.721729 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.721701    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/538d5103-1c92-48f1-acc7-f90956848727-log-socket\") pod \"ovnkube-node-22ldv\" (UID: \"538d5103-1c92-48f1-acc7-f90956848727\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ldv"
Oct 30 21:42:27.721729 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.721721    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cad67bc5-c8d2-497c-ba1b-cdbae5822cfa-installation-pull-secrets\") pod \"image-registry-b6cdbd45d-bdftd\" (UID: \"cad67bc5-c8d2-497c-ba1b-cdbae5822cfa\") " pod="openshift-image-registry/image-registry-b6cdbd45d-bdftd"
Oct 30 21:42:27.722089 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.721738    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l75p7\" (UniqueName: \"kubernetes.io/projected/28210ad9-18f0-4ecf-9bfc-b256440f0ce8-kube-api-access-l75p7\") pod \"ingress-canary-pzpgw\" (UID: \"28210ad9-18f0-4ecf-9bfc-b256440f0ce8\") " pod="openshift-ingress-canary/ingress-canary-pzpgw"
Oct 30 21:42:27.722089 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.721753    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ed08bc51-70e6-4658-adb3-ed36f5176552-host\") pod \"tuned-hskxk\" (UID: \"ed08bc51-70e6-4658-adb3-ed36f5176552\") " pod="openshift-cluster-node-tuning-operator/tuned-hskxk"
Oct 30 21:42:27.722089 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.721758    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/538d5103-1c92-48f1-acc7-f90956848727-ovn-node-metrics-cert\") pod \"ovnkube-node-22ldv\" (UID: \"538d5103-1c92-48f1-acc7-f90956848727\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ldv"
Oct 30 21:42:27.722089 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.721771    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a2e3b25e-b6dc-46ef-86e8-4e70f9dad200-multus-cni-dir\") pod \"multus-4nqkj\" (UID: \"a2e3b25e-b6dc-46ef-86e8-4e70f9dad200\") " pod="openshift-multus/multus-4nqkj"
Oct 30 21:42:27.722089 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.721796    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a2e3b25e-b6dc-46ef-86e8-4e70f9dad200-host-run-multus-certs\") pod \"multus-4nqkj\" (UID: \"a2e3b25e-b6dc-46ef-86e8-4e70f9dad200\") " pod="openshift-multus/multus-4nqkj"
Oct 30 21:42:27.722089 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.721865    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c69312da-8be3-4931-8a81-a8ff3f86d5e6-serving-cert\") pod \"insights-operator-7c4986d449-l9b9d\" (UID: \"c69312da-8be3-4931-8a81-a8ff3f86d5e6\") " pod="openshift-insights/insights-operator-7c4986d449-l9b9d"
Oct 30 21:42:27.722089 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.721900    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0126a0a0-a1c8-4db3-a6af-629360e247f4-serving-cert\") pod \"service-ca-operator-544d749c8c-vqwlt\" (UID: \"0126a0a0-a1c8-4db3-a6af-629360e247f4\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vqwlt"
Oct 30 21:42:27.722089 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.721971    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cad67bc5-c8d2-497c-ba1b-cdbae5822cfa-installation-pull-secrets\") pod \"image-registry-b6cdbd45d-bdftd\" (UID: \"cad67bc5-c8d2-497c-ba1b-cdbae5822cfa\") " pod="openshift-image-registry/image-registry-b6cdbd45d-bdftd"
Oct 30 21:42:27.722089 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.722005    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c69312da-8be3-4931-8a81-a8ff3f86d5e6-serving-cert\") pod \"insights-operator-7c4986d449-l9b9d\" (UID: \"c69312da-8be3-4931-8a81-a8ff3f86d5e6\") " pod="openshift-insights/insights-operator-7c4986d449-l9b9d"
Oct 30 21:42:27.722089 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.722029    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/538d5103-1c92-48f1-acc7-f90956848727-var-lib-openvswitch\") pod \"ovnkube-node-22ldv\" (UID: \"538d5103-1c92-48f1-acc7-f90956848727\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ldv"
Oct 30 21:42:27.722487 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.722098    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/538d5103-1c92-48f1-acc7-f90956848727-node-log\") pod \"ovnkube-node-22ldv\" (UID: \"538d5103-1c92-48f1-acc7-f90956848727\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ldv"
Oct 30 21:42:27.722487 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.722120    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/613e7c8a-e2c7-424a-93e0-7fa86d18a8a9-installation-pull-secrets\") pod \"image-registry-d44c8cbb7-lmw97\" (UID: \"613e7c8a-e2c7-424a-93e0-7fa86d18a8a9\") " pod="openshift-image-registry/image-registry-d44c8cbb7-lmw97"
Oct 30 21:42:27.722487 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.722141    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ed08bc51-70e6-4658-adb3-ed36f5176552-etc-sysconfig\") pod \"tuned-hskxk\" (UID: \"ed08bc51-70e6-4658-adb3-ed36f5176552\") " pod="openshift-cluster-node-tuning-operator/tuned-hskxk"
Oct 30 21:42:27.722487 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.722156    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ed08bc51-70e6-4658-adb3-ed36f5176552-run\") pod \"tuned-hskxk\" (UID: \"ed08bc51-70e6-4658-adb3-ed36f5176552\") " pod="openshift-cluster-node-tuning-operator/tuned-hskxk"
Oct 30 21:42:27.722487 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.722174    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a2e3b25e-b6dc-46ef-86e8-4e70f9dad200-cni-binary-copy\") pod \"multus-4nqkj\" (UID: \"a2e3b25e-b6dc-46ef-86e8-4e70f9dad200\") " pod="openshift-multus/multus-4nqkj"
Oct 30 21:42:27.722487 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.722200    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/538d5103-1c92-48f1-acc7-f90956848727-host-cni-bin\") pod \"ovnkube-node-22ldv\" (UID: \"538d5103-1c92-48f1-acc7-f90956848727\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ldv"
Oct 30 21:42:27.722487 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.722251    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0126a0a0-a1c8-4db3-a6af-629360e247f4-serving-cert\") pod \"service-ca-operator-544d749c8c-vqwlt\" (UID: \"0126a0a0-a1c8-4db3-a6af-629360e247f4\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vqwlt"
Oct 30 21:42:27.722487 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.722282    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/613e7c8a-e2c7-424a-93e0-7fa86d18a8a9-registry-tls\") pod \"image-registry-d44c8cbb7-lmw97\" (UID: \"613e7c8a-e2c7-424a-93e0-7fa86d18a8a9\") " pod="openshift-image-registry/image-registry-d44c8cbb7-lmw97"
Oct 30 21:42:27.722487 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.722323    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ed08bc51-70e6-4658-adb3-ed36f5176552-var-lib-kubelet\") pod \"tuned-hskxk\" (UID: \"ed08bc51-70e6-4658-adb3-ed36f5176552\") " pod="openshift-cluster-node-tuning-operator/tuned-hskxk"
Oct 30 21:42:27.722487 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.722359    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a2e3b25e-b6dc-46ef-86e8-4e70f9dad200-cni-binary-copy\") pod \"multus-4nqkj\" (UID: \"a2e3b25e-b6dc-46ef-86e8-4e70f9dad200\") " pod="openshift-multus/multus-4nqkj"
Oct 30 21:42:27.722487 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.722362    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pd6bv\" (UniqueName: \"kubernetes.io/projected/ed08bc51-70e6-4658-adb3-ed36f5176552-kube-api-access-pd6bv\") pod \"tuned-hskxk\" (UID: \"ed08bc51-70e6-4658-adb3-ed36f5176552\") " pod="openshift-cluster-node-tuning-operator/tuned-hskxk"
Oct 30 21:42:27.722487 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.722373    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/613e7c8a-e2c7-424a-93e0-7fa86d18a8a9-installation-pull-secrets\") pod \"image-registry-d44c8cbb7-lmw97\" (UID: \"613e7c8a-e2c7-424a-93e0-7fa86d18a8a9\") " pod="openshift-image-registry/image-registry-d44c8cbb7-lmw97"
Oct 30 21:42:27.722487 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.722389    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a2e3b25e-b6dc-46ef-86e8-4e70f9dad200-host-var-lib-cni-bin\") pod \"multus-4nqkj\" (UID: \"a2e3b25e-b6dc-46ef-86e8-4e70f9dad200\") " pod="openshift-multus/multus-4nqkj"
Oct 30 21:42:27.722487 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.722407    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nc76p\" (UniqueName: \"kubernetes.io/projected/4427f59d-8003-440f-9646-a46b176cc4a7-kube-api-access-nc76p\") pod \"aws-ebs-csi-driver-node-qk9dg\" (UID: \"4427f59d-8003-440f-9646-a46b176cc4a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qk9dg"
Oct 30 21:42:27.722487 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.722451    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1f3b6a23-d7ea-4ce4-b1e8-68133c4b4ebc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-b47v2\" (UID: \"1f3b6a23-d7ea-4ce4-b1e8-68133c4b4ebc\") " pod="openshift-multus/multus-additional-cni-plugins-b47v2"
Oct 30 21:42:27.723044 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.722563    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xk5zb\" (UniqueName: \"kubernetes.io/projected/c798fd57-e0be-4b61-860b-0b833aca7784-kube-api-access-xk5zb\") pod \"kas-connection-checker-685895c7c9-kz6dl\" (UID: \"c798fd57-e0be-4b61-860b-0b833aca7784\") " pod="kube-system/kas-connection-checker-685895c7c9-kz6dl"
Oct 30 21:42:27.723044 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.722581    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1f3b6a23-d7ea-4ce4-b1e8-68133c4b4ebc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-b47v2\" (UID: \"1f3b6a23-d7ea-4ce4-b1e8-68133c4b4ebc\") " pod="openshift-multus/multus-additional-cni-plugins-b47v2"
Oct 30 21:42:27.723044 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.722616    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/538d5103-1c92-48f1-acc7-f90956848727-ovnkube-script-lib\") pod \"ovnkube-node-22ldv\" (UID: \"538d5103-1c92-48f1-acc7-f90956848727\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ldv"
Oct 30 21:42:27.723044 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.722706    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cad67bc5-c8d2-497c-ba1b-cdbae5822cfa-ca-trust-extracted\") pod \"image-registry-b6cdbd45d-bdftd\" (UID: \"cad67bc5-c8d2-497c-ba1b-cdbae5822cfa\") " pod="openshift-image-registry/image-registry-b6cdbd45d-bdftd"
Oct 30 21:42:27.723044 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.722727    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f648e933-f035-486a-88d5-1b185c89782e-nginx-conf\") pod \"networking-console-plugin-86c478bb88-j5xsr\" (UID: \"f648e933-f035-486a-88d5-1b185c89782e\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-j5xsr"
Oct 30 21:42:27.723044 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.722744    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kr7ww\" (UniqueName: \"kubernetes.io/projected/20f55ce3-5b04-4916-b529-0dd596aba68f-kube-api-access-kr7ww\") pod \"network-check-source-6f78d59f99-rmpxf\" (UID: \"20f55ce3-5b04-4916-b529-0dd596aba68f\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-rmpxf"
Oct 30 21:42:27.723044 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.722763    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4427f59d-8003-440f-9646-a46b176cc4a7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qk9dg\" (UID: \"4427f59d-8003-440f-9646-a46b176cc4a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qk9dg"
Oct 30 21:42:27.723044 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.722773    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cad67bc5-c8d2-497c-ba1b-cdbae5822cfa-ca-trust-extracted\") pod \"image-registry-b6cdbd45d-bdftd\" (UID: \"cad67bc5-c8d2-497c-ba1b-cdbae5822cfa\") " pod="openshift-image-registry/image-registry-b6cdbd45d-bdftd"
Oct 30 21:42:27.723044 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.722777    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/74ff988a-c05e-4a3e-adff-5945b79cf248-crio-socket\") pod \"insights-runtime-extractor-mstdj\" (UID: \"74ff988a-c05e-4a3e-adff-5945b79cf248\") " pod="openshift-insights/insights-runtime-extractor-mstdj"
Oct 30 21:42:27.723044 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.722788    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1f3b6a23-d7ea-4ce4-b1e8-68133c4b4ebc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-b47v2\" (UID: \"1f3b6a23-d7ea-4ce4-b1e8-68133c4b4ebc\") " pod="openshift-multus/multus-additional-cni-plugins-b47v2"
Oct 30 21:42:27.723044 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.722813    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cad67bc5-c8d2-497c-ba1b-cdbae5822cfa-trusted-ca\") pod \"image-registry-b6cdbd45d-bdftd\" (UID: \"cad67bc5-c8d2-497c-ba1b-cdbae5822cfa\") " pod="openshift-image-registry/image-registry-b6cdbd45d-bdftd"
Oct 30 21:42:27.723044 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.722844    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0126a0a0-a1c8-4db3-a6af-629360e247f4-config\") pod \"service-ca-operator-544d749c8c-vqwlt\" (UID: \"0126a0a0-a1c8-4db3-a6af-629360e247f4\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vqwlt"
Oct 30 21:42:27.723044 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.722848    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/538d5103-1c92-48f1-acc7-f90956848727-ovnkube-script-lib\") pod \"ovnkube-node-22ldv\" (UID: \"538d5103-1c92-48f1-acc7-f90956848727\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ldv"
Oct 30 21:42:27.723044 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.722872    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2205e8e6-f038-47fa-b393-0abe4d622e03-metrics-certs\") pod \"network-metrics-daemon-q6tjc\" (UID: \"2205e8e6-f038-47fa-b393-0abe4d622e03\") " pod="openshift-multus/network-metrics-daemon-q6tjc"
Oct 30 21:42:27.723044 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.722963    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/538d5103-1c92-48f1-acc7-f90956848727-ovnkube-config\") pod \"ovnkube-node-22ldv\" (UID: \"538d5103-1c92-48f1-acc7-f90956848727\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ldv"
Oct 30 21:42:27.723044 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.723002    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/c2386477-903e-4c6c-b807-3f473a58a6b6-signing-cabundle\") pod \"service-ca-5fffc97bdf-th574\" (UID: \"c2386477-903e-4c6c-b807-3f473a58a6b6\") " pod="openshift-service-ca/service-ca-5fffc97bdf-th574"
Oct 30 21:42:27.723044 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.723037    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a2e3b25e-b6dc-46ef-86e8-4e70f9dad200-host-run-k8s-cni-cncf-io\") pod \"multus-4nqkj\" (UID: \"a2e3b25e-b6dc-46ef-86e8-4e70f9dad200\") " pod="openshift-multus/multus-4nqkj"
Oct 30 21:42:27.723659 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.723092    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0126a0a0-a1c8-4db3-a6af-629360e247f4-config\") pod \"service-ca-operator-544d749c8c-vqwlt\" (UID: \"0126a0a0-a1c8-4db3-a6af-629360e247f4\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vqwlt"
Oct 30 21:42:27.723659 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.723090    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f648e933-f035-486a-88d5-1b185c89782e-nginx-conf\") pod \"networking-console-plugin-86c478bb88-j5xsr\" (UID: \"f648e933-f035-486a-88d5-1b185c89782e\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-j5xsr"
Oct 30 21:42:27.723659 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.723226    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/538d5103-1c92-48f1-acc7-f90956848727-ovnkube-config\") pod \"ovnkube-node-22ldv\" (UID: \"538d5103-1c92-48f1-acc7-f90956848727\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ldv"
Oct 30 21:42:27.723659 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.723272    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/c2386477-903e-4c6c-b807-3f473a58a6b6-signing-cabundle\") pod \"service-ca-5fffc97bdf-th574\" (UID: \"c2386477-903e-4c6c-b807-3f473a58a6b6\") " pod="openshift-service-ca/service-ca-5fffc97bdf-th574"
Oct 30 21:42:27.723659 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.723513    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cad67bc5-c8d2-497c-ba1b-cdbae5822cfa-trusted-ca\") pod \"image-registry-b6cdbd45d-bdftd\" (UID: \"cad67bc5-c8d2-497c-ba1b-cdbae5822cfa\") " pod="openshift-image-registry/image-registry-b6cdbd45d-bdftd"
Oct 30 21:42:27.771783 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.771760    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-579v9\" (UniqueName: \"kubernetes.io/projected/3784e680-5efe-4ffb-ad71-956d05b57dce-kube-api-access-579v9\") pod \"network-check-target-979r9\" (UID: \"3784e680-5efe-4ffb-ad71-956d05b57dce\") " pod="openshift-network-diagnostics/network-check-target-979r9"
Oct 30 21:42:27.772027 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.771812    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fcz7\" (UniqueName: \"kubernetes.io/projected/102b8049-7349-4e2b-974a-44c901eebcf3-kube-api-access-5fcz7\") pod \"cluster-monitoring-operator-884576c57-ns759\" (UID: \"102b8049-7349-4e2b-974a-44c901eebcf3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ns759"
Oct 30 21:42:27.772216 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.772199    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mms6q\" (UniqueName: \"kubernetes.io/projected/84d9b682-3241-40af-87d5-f87d54ce3697-kube-api-access-mms6q\") pod \"dns-default-6pttx\" (UID: \"84d9b682-3241-40af-87d5-f87d54ce3697\") " pod="openshift-dns/dns-default-6pttx"
Oct 30 21:42:27.772390 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.772370    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhbrj\" (UniqueName: \"kubernetes.io/projected/d700d1eb-fbf4-45df-9b57-eca08d87ebe2-kube-api-access-nhbrj\") pod \"volume-data-source-validator-58c9f5f46d-s6427\" (UID: \"d700d1eb-fbf4-45df-9b57-eca08d87ebe2\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-s6427"
Oct 30 21:42:27.772446 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.772431    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4658n\" (UniqueName: \"kubernetes.io/projected/879022d4-06b9-44cc-a2a2-dc307b27345e-kube-api-access-4658n\") pod \"migrator-66df5d496b-rznzd\" (UID: \"879022d4-06b9-44cc-a2a2-dc307b27345e\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-rznzd"
Oct 30 21:42:27.772750 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.772735    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpbtv\" (UniqueName: \"kubernetes.io/projected/b227ad6a-9e59-415e-b7e6-f7c25e988abb-kube-api-access-qpbtv\") pod \"node-ca-x8gnj\" (UID: \"b227ad6a-9e59-415e-b7e6-f7c25e988abb\") " pod="openshift-image-registry/node-ca-x8gnj"
Oct 30 21:42:27.772869 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.772855    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8w7g2\" (UniqueName: \"kubernetes.io/projected/798f248f-d429-4f56-8752-e157dd94206b-kube-api-access-8w7g2\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-mgxcj\" (UID: \"798f248f-d429-4f56-8752-e157dd94206b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-mgxcj"
Oct 30 21:42:27.776487 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.776468    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srwt7\" (UniqueName: \"kubernetes.io/projected/0126a0a0-a1c8-4db3-a6af-629360e247f4-kube-api-access-srwt7\") pod \"service-ca-operator-544d749c8c-vqwlt\" (UID: \"0126a0a0-a1c8-4db3-a6af-629360e247f4\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vqwlt"
Oct 30 21:42:27.777255 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.776808    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-424gm\" (UniqueName: \"kubernetes.io/projected/a32aacdc-53be-41cf-839e-435d43361890-kube-api-access-424gm\") pod \"kas-connection-checker-685895c7c9-cklcw\" (UID: \"a32aacdc-53be-41cf-839e-435d43361890\") " pod="kube-system/kas-connection-checker-685895c7c9-cklcw"
Oct 30 21:42:27.777255 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.776847    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cad67bc5-c8d2-497c-ba1b-cdbae5822cfa-bound-sa-token\") pod \"image-registry-b6cdbd45d-bdftd\" (UID: \"cad67bc5-c8d2-497c-ba1b-cdbae5822cfa\") " pod="openshift-image-registry/image-registry-b6cdbd45d-bdftd"
Oct 30 21:42:27.777255 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.776852    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrpz5\" (UniqueName: \"kubernetes.io/projected/538d5103-1c92-48f1-acc7-f90956848727-kube-api-access-hrpz5\") pod \"ovnkube-node-22ldv\" (UID: \"538d5103-1c92-48f1-acc7-f90956848727\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ldv"
Oct 30 21:42:27.777255 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.776877    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qd5b8\" (UniqueName: \"kubernetes.io/projected/c69312da-8be3-4931-8a81-a8ff3f86d5e6-kube-api-access-qd5b8\") pod \"insights-operator-7c4986d449-l9b9d\" (UID: \"c69312da-8be3-4931-8a81-a8ff3f86d5e6\") " pod="openshift-insights/insights-operator-7c4986d449-l9b9d"
Oct 30 21:42:27.777255 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.777209    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbwzr\" (UniqueName: \"kubernetes.io/projected/2205e8e6-f038-47fa-b393-0abe4d622e03-kube-api-access-rbwzr\") pod \"network-metrics-daemon-q6tjc\" (UID: \"2205e8e6-f038-47fa-b393-0abe4d622e03\") " pod="openshift-multus/network-metrics-daemon-q6tjc"
Oct 30 21:42:27.777443 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.777253    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xm5l5\" (UniqueName: \"kubernetes.io/projected/101c697b-d7b8-4cfe-9d4c-818b5c04108f-kube-api-access-xm5l5\") pod \"periodic-gathering-7t4kc-b65p7\" (UID: \"101c697b-d7b8-4cfe-9d4c-818b5c04108f\") " pod="openshift-insights/periodic-gathering-7t4kc-b65p7"
Oct 30 21:42:27.777443 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.777324    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9grk\" (UniqueName: \"kubernetes.io/projected/033356b5-3c05-4f49-ad6a-7c369259970d-kube-api-access-d9grk\") pod \"migrator-66df5d496b-q4khr\" (UID: \"033356b5-3c05-4f49-ad6a-7c369259970d\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-q4khr"
Oct 30 21:42:27.780633 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.780616    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdtc5\" (UniqueName: \"kubernetes.io/projected/3f97ee34-5dfc-454e-ba5c-afd939f3bd25-kube-api-access-cdtc5\") pod \"kas-connection-checker-685895c7c9-bgnkz\" (UID: \"3f97ee34-5dfc-454e-ba5c-afd939f3bd25\") " pod="kube-system/kas-connection-checker-685895c7c9-bgnkz"
Oct 30 21:42:27.780921 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.780888    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4tb7\" (UniqueName: \"kubernetes.io/projected/9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d-kube-api-access-n4tb7\") pod \"router-default-6c7f96b45-4wj99\" (UID: \"9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d\") " pod="openshift-ingress/router-default-6c7f96b45-4wj99"
Oct 30 21:42:27.781110 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.781095    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zv2qw\" (UniqueName: \"kubernetes.io/projected/c2386477-903e-4c6c-b807-3f473a58a6b6-kube-api-access-zv2qw\") pod \"service-ca-5fffc97bdf-th574\" (UID: \"c2386477-903e-4c6c-b807-3f473a58a6b6\") " pod="openshift-service-ca/service-ca-5fffc97bdf-th574"
Oct 30 21:42:27.781236 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.781215    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swjcw\" (UniqueName: \"kubernetes.io/projected/0929b9f6-12ec-4f4d-8ba3-5cef9defba15-kube-api-access-swjcw\") pod \"iptables-alerter-wbq4l\" (UID: \"0929b9f6-12ec-4f4d-8ba3-5cef9defba15\") " pod="openshift-network-operator/iptables-alerter-wbq4l"
Oct 30 21:42:27.781309 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.781293    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hkfh\" (UniqueName: \"kubernetes.io/projected/71ddf455-cbf7-49a6-9773-30f7e4997a39-kube-api-access-7hkfh\") pod \"node-resolver-2wsrr\" (UID: \"71ddf455-cbf7-49a6-9773-30f7e4997a39\") " pod="openshift-dns/node-resolver-2wsrr"
Oct 30 21:42:27.781434 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.781418    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdbmm\" (UniqueName: \"kubernetes.io/projected/0bb51175-fa5e-4472-8d83-106bca721585-kube-api-access-pdbmm\") pod \"cluster-samples-operator-6ddb6c8b94-tlwms\" (UID: \"0bb51175-fa5e-4472-8d83-106bca721585\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-tlwms"
Oct 30 21:42:27.782395 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.782378    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hw226\" (UniqueName: \"kubernetes.io/projected/613e7c8a-e2c7-424a-93e0-7fa86d18a8a9-kube-api-access-hw226\") pod \"image-registry-d44c8cbb7-lmw97\" (UID: \"613e7c8a-e2c7-424a-93e0-7fa86d18a8a9\") " pod="openshift-image-registry/image-registry-d44c8cbb7-lmw97"
Oct 30 21:42:27.782466 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.782452    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67qtb\" (UniqueName: \"kubernetes.io/projected/74ff988a-c05e-4a3e-adff-5945b79cf248-kube-api-access-67qtb\") pod \"insights-runtime-extractor-mstdj\" (UID: \"74ff988a-c05e-4a3e-adff-5945b79cf248\") " pod="openshift-insights/insights-runtime-extractor-mstdj"
Oct 30 21:42:27.783036 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.782998    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8b7v\" (UniqueName: \"kubernetes.io/projected/a2e3b25e-b6dc-46ef-86e8-4e70f9dad200-kube-api-access-p8b7v\") pod \"multus-4nqkj\" (UID: \"a2e3b25e-b6dc-46ef-86e8-4e70f9dad200\") " pod="openshift-multus/multus-4nqkj"
Oct 30 21:42:27.783163 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.783137    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l75p7\" (UniqueName: \"kubernetes.io/projected/28210ad9-18f0-4ecf-9bfc-b256440f0ce8-kube-api-access-l75p7\") pod \"ingress-canary-pzpgw\" (UID: \"28210ad9-18f0-4ecf-9bfc-b256440f0ce8\") " pod="openshift-ingress-canary/ingress-canary-pzpgw"
Oct 30 21:42:27.783424 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.783405    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nc76p\" (UniqueName: \"kubernetes.io/projected/4427f59d-8003-440f-9646-a46b176cc4a7-kube-api-access-nc76p\") pod \"aws-ebs-csi-driver-node-qk9dg\" (UID: \"4427f59d-8003-440f-9646-a46b176cc4a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qk9dg"
Oct 30 21:42:27.783624 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.783605    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9dpv\" (UniqueName: \"kubernetes.io/projected/cad67bc5-c8d2-497c-ba1b-cdbae5822cfa-kube-api-access-m9dpv\") pod \"image-registry-b6cdbd45d-bdftd\" (UID: \"cad67bc5-c8d2-497c-ba1b-cdbae5822cfa\") " pod="openshift-image-registry/image-registry-b6cdbd45d-bdftd"
Oct 30 21:42:27.783690 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.783666    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gq9m\" (UniqueName: \"kubernetes.io/projected/1f3b6a23-d7ea-4ce4-b1e8-68133c4b4ebc-kube-api-access-7gq9m\") pod \"multus-additional-cni-plugins-b47v2\" (UID: \"1f3b6a23-d7ea-4ce4-b1e8-68133c4b4ebc\") " pod="openshift-multus/multus-additional-cni-plugins-b47v2"
Oct 30 21:42:27.783878 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.783862    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqmr4\" (UniqueName: \"kubernetes.io/projected/5e74071e-0462-42b5-a05c-d90bb638478a-kube-api-access-vqmr4\") pod \"console-operator-7cd56b8dbc-sclrt\" (UID: \"5e74071e-0462-42b5-a05c-d90bb638478a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-sclrt"
Oct 30 21:42:27.784308 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.784292    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/613e7c8a-e2c7-424a-93e0-7fa86d18a8a9-bound-sa-token\") pod \"image-registry-d44c8cbb7-lmw97\" (UID: \"613e7c8a-e2c7-424a-93e0-7fa86d18a8a9\") " pod="openshift-image-registry/image-registry-d44c8cbb7-lmw97"
Oct 30 21:42:27.808182 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.808161    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pd6bv\" (UniqueName: \"kubernetes.io/projected/ed08bc51-70e6-4658-adb3-ed36f5176552-kube-api-access-pd6bv\") pod \"tuned-hskxk\" (UID: \"ed08bc51-70e6-4658-adb3-ed36f5176552\") " pod="openshift-cluster-node-tuning-operator/tuned-hskxk"
Oct 30 21:42:27.808256 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.808168    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xk5zb\" (UniqueName: \"kubernetes.io/projected/c798fd57-e0be-4b61-860b-0b833aca7784-kube-api-access-xk5zb\") pod \"kas-connection-checker-685895c7c9-kz6dl\" (UID: \"c798fd57-e0be-4b61-860b-0b833aca7784\") " pod="kube-system/kas-connection-checker-685895c7c9-kz6dl"
Oct 30 21:42:27.808472 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.808455    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kr7ww\" (UniqueName: \"kubernetes.io/projected/20f55ce3-5b04-4916-b529-0dd596aba68f-kube-api-access-kr7ww\") pod \"network-check-source-6f78d59f99-rmpxf\" (UID: \"20f55ce3-5b04-4916-b529-0dd596aba68f\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-rmpxf"
Oct 30 21:42:27.824067 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.824046    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/538d5103-1c92-48f1-acc7-f90956848727-systemd-units\") pod \"ovnkube-node-22ldv\" (UID: \"538d5103-1c92-48f1-acc7-f90956848727\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ldv"
Oct 30 21:42:27.824141 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.824070    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a2e3b25e-b6dc-46ef-86e8-4e70f9dad200-multus-socket-dir-parent\") pod \"multus-4nqkj\" (UID: \"a2e3b25e-b6dc-46ef-86e8-4e70f9dad200\") " pod="openshift-multus/multus-4nqkj"
Oct 30 21:42:27.824141 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.824087    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/74ff988a-c05e-4a3e-adff-5945b79cf248-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mstdj\" (UID: \"74ff988a-c05e-4a3e-adff-5945b79cf248\") " pod="openshift-insights/insights-runtime-extractor-mstdj"
Oct 30 21:42:27.824141 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.824102    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/102b8049-7349-4e2b-974a-44c901eebcf3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-ns759\" (UID: \"102b8049-7349-4e2b-974a-44c901eebcf3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ns759"
Oct 30 21:42:27.824226 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.824157    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/538d5103-1c92-48f1-acc7-f90956848727-systemd-units\") pod \"ovnkube-node-22ldv\" (UID: \"538d5103-1c92-48f1-acc7-f90956848727\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ldv"
Oct 30 21:42:27.824226 ip-10-0-130-60 kubenswrapper[5061]: E1030 21:42:27.824178    5061 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 21:42:27.824317 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.824158    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a2e3b25e-b6dc-46ef-86e8-4e70f9dad200-multus-socket-dir-parent\") pod \"multus-4nqkj\" (UID: \"a2e3b25e-b6dc-46ef-86e8-4e70f9dad200\") " pod="openshift-multus/multus-4nqkj"
Oct 30 21:42:27.824317 ip-10-0-130-60 kubenswrapper[5061]: E1030 21:42:27.824245    5061 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74ff988a-c05e-4a3e-adff-5945b79cf248-insights-runtime-extractor-tls podName:74ff988a-c05e-4a3e-adff-5945b79cf248 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:28.324226937 +0000 UTC m=+1.936996689 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/74ff988a-c05e-4a3e-adff-5945b79cf248-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mstdj" (UID: "74ff988a-c05e-4a3e-adff-5945b79cf248") : secret "insights-runtime-extractor-tls" not found
Oct 30 21:42:27.824317 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.824221    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ed08bc51-70e6-4658-adb3-ed36f5176552-lib-modules\") pod \"tuned-hskxk\" (UID: \"ed08bc51-70e6-4658-adb3-ed36f5176552\") " pod="openshift-cluster-node-tuning-operator/tuned-hskxk"
Oct 30 21:42:27.824317 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.824279    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d-service-ca-bundle\") pod \"router-default-6c7f96b45-4wj99\" (UID: \"9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d\") " pod="openshift-ingress/router-default-6c7f96b45-4wj99"
Oct 30 21:42:27.824317 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.824298    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ed08bc51-70e6-4658-adb3-ed36f5176552-etc-sysctl-conf\") pod \"tuned-hskxk\" (UID: \"ed08bc51-70e6-4658-adb3-ed36f5176552\") " pod="openshift-cluster-node-tuning-operator/tuned-hskxk"
Oct 30 21:42:27.824317 ip-10-0-130-60 kubenswrapper[5061]: E1030 21:42:27.824305    5061 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 21:42:27.824656 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.824318    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/71ddf455-cbf7-49a6-9773-30f7e4997a39-hosts-file\") pod \"node-resolver-2wsrr\" (UID: \"71ddf455-cbf7-49a6-9773-30f7e4997a39\") " pod="openshift-dns/node-resolver-2wsrr"
Oct 30 21:42:27.824656 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.824328    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ed08bc51-70e6-4658-adb3-ed36f5176552-lib-modules\") pod \"tuned-hskxk\" (UID: \"ed08bc51-70e6-4658-adb3-ed36f5176552\") " pod="openshift-cluster-node-tuning-operator/tuned-hskxk"
Oct 30 21:42:27.824656 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.824340    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a2e3b25e-b6dc-46ef-86e8-4e70f9dad200-cnibin\") pod \"multus-4nqkj\" (UID: \"a2e3b25e-b6dc-46ef-86e8-4e70f9dad200\") " pod="openshift-multus/multus-4nqkj"
Oct 30 21:42:27.824656 ip-10-0-130-60 kubenswrapper[5061]: E1030 21:42:27.824360    5061 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/102b8049-7349-4e2b-974a-44c901eebcf3-cluster-monitoring-operator-tls podName:102b8049-7349-4e2b-974a-44c901eebcf3 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:28.324345781 +0000 UTC m=+1.937115530 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/102b8049-7349-4e2b-974a-44c901eebcf3-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-ns759" (UID: "102b8049-7349-4e2b-974a-44c901eebcf3") : secret "cluster-monitoring-operator-tls" not found
Oct 30 21:42:27.824656 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.824371    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a2e3b25e-b6dc-46ef-86e8-4e70f9dad200-cnibin\") pod \"multus-4nqkj\" (UID: \"a2e3b25e-b6dc-46ef-86e8-4e70f9dad200\") " pod="openshift-multus/multus-4nqkj"
Oct 30 21:42:27.824656 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.824389    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4427f59d-8003-440f-9646-a46b176cc4a7-etc-selinux\") pod \"aws-ebs-csi-driver-node-qk9dg\" (UID: \"4427f59d-8003-440f-9646-a46b176cc4a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qk9dg"
Oct 30 21:42:27.824656 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.824420    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fe41c70b-7e2c-432f-8389-0282d6fdb332-kubelet-config\") pod \"global-pull-secret-syncer-6wbw4\" (UID: \"fe41c70b-7e2c-432f-8389-0282d6fdb332\") " pod="kube-system/global-pull-secret-syncer-6wbw4"
Oct 30 21:42:27.824656 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.824433    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4427f59d-8003-440f-9646-a46b176cc4a7-etc-selinux\") pod \"aws-ebs-csi-driver-node-qk9dg\" (UID: \"4427f59d-8003-440f-9646-a46b176cc4a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qk9dg"
Oct 30 21:42:27.824656 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.824437    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/538d5103-1c92-48f1-acc7-f90956848727-host-slash\") pod \"ovnkube-node-22ldv\" (UID: \"538d5103-1c92-48f1-acc7-f90956848727\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ldv"
Oct 30 21:42:27.824656 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.824460    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/538d5103-1c92-48f1-acc7-f90956848727-host-slash\") pod \"ovnkube-node-22ldv\" (UID: \"538d5103-1c92-48f1-acc7-f90956848727\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ldv"
Oct 30 21:42:27.824656 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.824470    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/538d5103-1c92-48f1-acc7-f90956848727-host-run-ovn-kubernetes\") pod \"ovnkube-node-22ldv\" (UID: \"538d5103-1c92-48f1-acc7-f90956848727\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ldv"
Oct 30 21:42:27.824656 ip-10-0-130-60 kubenswrapper[5061]: E1030 21:42:27.824491    5061 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d-service-ca-bundle podName:9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d nodeName:}" failed. No retries permitted until 2026-10-30 21:42:28.324474584 +0000 UTC m=+1.937244335 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d-service-ca-bundle") pod "router-default-6c7f96b45-4wj99" (UID: "9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d") : configmap references non-existent config key: service-ca.crt
Oct 30 21:42:27.824656 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.824495    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fe41c70b-7e2c-432f-8389-0282d6fdb332-kubelet-config\") pod \"global-pull-secret-syncer-6wbw4\" (UID: \"fe41c70b-7e2c-432f-8389-0282d6fdb332\") " pod="kube-system/global-pull-secret-syncer-6wbw4"
Oct 30 21:42:27.824656 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.824495    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/538d5103-1c92-48f1-acc7-f90956848727-host-run-ovn-kubernetes\") pod \"ovnkube-node-22ldv\" (UID: \"538d5103-1c92-48f1-acc7-f90956848727\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ldv"
Oct 30 21:42:27.824656 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.824547    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/71ddf455-cbf7-49a6-9773-30f7e4997a39-hosts-file\") pod \"node-resolver-2wsrr\" (UID: \"71ddf455-cbf7-49a6-9773-30f7e4997a39\") " pod="openshift-dns/node-resolver-2wsrr"
Oct 30 21:42:27.824656 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.824552    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a2e3b25e-b6dc-46ef-86e8-4e70f9dad200-host-run-netns\") pod \"multus-4nqkj\" (UID: \"a2e3b25e-b6dc-46ef-86e8-4e70f9dad200\") " pod="openshift-multus/multus-4nqkj"
Oct 30 21:42:27.824656 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.824586    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a2e3b25e-b6dc-46ef-86e8-4e70f9dad200-system-cni-dir\") pod \"multus-4nqkj\" (UID: \"a2e3b25e-b6dc-46ef-86e8-4e70f9dad200\") " pod="openshift-multus/multus-4nqkj"
Oct 30 21:42:27.824656 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.824602    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ed08bc51-70e6-4658-adb3-ed36f5176552-etc-sysctl-conf\") pod \"tuned-hskxk\" (UID: \"ed08bc51-70e6-4658-adb3-ed36f5176552\") " pod="openshift-cluster-node-tuning-operator/tuned-hskxk"
Oct 30 21:42:27.824656 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.824611    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a2e3b25e-b6dc-46ef-86e8-4e70f9dad200-etc-kubernetes\") pod \"multus-4nqkj\" (UID: \"a2e3b25e-b6dc-46ef-86e8-4e70f9dad200\") " pod="openshift-multus/multus-4nqkj"
Oct 30 21:42:27.824656 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.824596    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a2e3b25e-b6dc-46ef-86e8-4e70f9dad200-host-run-netns\") pod \"multus-4nqkj\" (UID: \"a2e3b25e-b6dc-46ef-86e8-4e70f9dad200\") " pod="openshift-multus/multus-4nqkj"
Oct 30 21:42:27.824656 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.824639    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a2e3b25e-b6dc-46ef-86e8-4e70f9dad200-system-cni-dir\") pod \"multus-4nqkj\" (UID: \"a2e3b25e-b6dc-46ef-86e8-4e70f9dad200\") " pod="openshift-multus/multus-4nqkj"
Oct 30 21:42:27.824656 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.824639    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/538d5103-1c92-48f1-acc7-f90956848727-run-openvswitch\") pod \"ovnkube-node-22ldv\" (UID: \"538d5103-1c92-48f1-acc7-f90956848727\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ldv"
Oct 30 21:42:27.824656 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.824649    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a2e3b25e-b6dc-46ef-86e8-4e70f9dad200-etc-kubernetes\") pod \"multus-4nqkj\" (UID: \"a2e3b25e-b6dc-46ef-86e8-4e70f9dad200\") " pod="openshift-multus/multus-4nqkj"
Oct 30 21:42:27.824656 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.824662    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ed08bc51-70e6-4658-adb3-ed36f5176552-etc-modprobe-d\") pod \"tuned-hskxk\" (UID: \"ed08bc51-70e6-4658-adb3-ed36f5176552\") " pod="openshift-cluster-node-tuning-operator/tuned-hskxk"
Oct 30 21:42:27.825620 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.824673    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/538d5103-1c92-48f1-acc7-f90956848727-run-openvswitch\") pod \"ovnkube-node-22ldv\" (UID: \"538d5103-1c92-48f1-acc7-f90956848727\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ldv"
Oct 30 21:42:27.825620 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.824678    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b227ad6a-9e59-415e-b7e6-f7c25e988abb-host\") pod \"node-ca-x8gnj\" (UID: \"b227ad6a-9e59-415e-b7e6-f7c25e988abb\") " pod="openshift-image-registry/node-ca-x8gnj"
Oct 30 21:42:27.825620 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.824716    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b227ad6a-9e59-415e-b7e6-f7c25e988abb-host\") pod \"node-ca-x8gnj\" (UID: \"b227ad6a-9e59-415e-b7e6-f7c25e988abb\") " pod="openshift-image-registry/node-ca-x8gnj"
Oct 30 21:42:27.825620 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.824724    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4427f59d-8003-440f-9646-a46b176cc4a7-device-dir\") pod \"aws-ebs-csi-driver-node-qk9dg\" (UID: \"4427f59d-8003-440f-9646-a46b176cc4a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qk9dg"
Oct 30 21:42:27.825620 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.824743    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ed08bc51-70e6-4658-adb3-ed36f5176552-etc-modprobe-d\") pod \"tuned-hskxk\" (UID: \"ed08bc51-70e6-4658-adb3-ed36f5176552\") " pod="openshift-cluster-node-tuning-operator/tuned-hskxk"
Oct 30 21:42:27.825620 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.824745    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/538d5103-1c92-48f1-acc7-f90956848727-host-cni-netd\") pod \"ovnkube-node-22ldv\" (UID: \"538d5103-1c92-48f1-acc7-f90956848727\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ldv"
Oct 30 21:42:27.825620 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.824765    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fe41c70b-7e2c-432f-8389-0282d6fdb332-dbus\") pod \"global-pull-secret-syncer-6wbw4\" (UID: \"fe41c70b-7e2c-432f-8389-0282d6fdb332\") " pod="kube-system/global-pull-secret-syncer-6wbw4"
Oct 30 21:42:27.825620 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.824768    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4427f59d-8003-440f-9646-a46b176cc4a7-device-dir\") pod \"aws-ebs-csi-driver-node-qk9dg\" (UID: \"4427f59d-8003-440f-9646-a46b176cc4a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qk9dg"
Oct 30 21:42:27.825620 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.824782    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a2e3b25e-b6dc-46ef-86e8-4e70f9dad200-host-var-lib-cni-multus\") pod \"multus-4nqkj\" (UID: \"a2e3b25e-b6dc-46ef-86e8-4e70f9dad200\") " pod="openshift-multus/multus-4nqkj"
Oct 30 21:42:27.825620 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.824806    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a2e3b25e-b6dc-46ef-86e8-4e70f9dad200-multus-conf-dir\") pod \"multus-4nqkj\" (UID: \"a2e3b25e-b6dc-46ef-86e8-4e70f9dad200\") " pod="openshift-multus/multus-4nqkj"
Oct 30 21:42:27.825620 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.824818    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/538d5103-1c92-48f1-acc7-f90956848727-host-cni-netd\") pod \"ovnkube-node-22ldv\" (UID: \"538d5103-1c92-48f1-acc7-f90956848727\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ldv"
Oct 30 21:42:27.825620 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.824830    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/538d5103-1c92-48f1-acc7-f90956848727-run-ovn\") pod \"ovnkube-node-22ldv\" (UID: \"538d5103-1c92-48f1-acc7-f90956848727\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ldv"
Oct 30 21:42:27.825620 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.824854    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fe41c70b-7e2c-432f-8389-0282d6fdb332-dbus\") pod \"global-pull-secret-syncer-6wbw4\" (UID: \"fe41c70b-7e2c-432f-8389-0282d6fdb332\") " pod="kube-system/global-pull-secret-syncer-6wbw4"
Oct 30 21:42:27.825620 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.824854    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/538d5103-1c92-48f1-acc7-f90956848727-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-22ldv\" (UID: \"538d5103-1c92-48f1-acc7-f90956848727\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ldv"
Oct 30 21:42:27.825620 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.824862    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a2e3b25e-b6dc-46ef-86e8-4e70f9dad200-host-var-lib-cni-multus\") pod \"multus-4nqkj\" (UID: \"a2e3b25e-b6dc-46ef-86e8-4e70f9dad200\") " pod="openshift-multus/multus-4nqkj"
Oct 30 21:42:27.825620 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.824857    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a2e3b25e-b6dc-46ef-86e8-4e70f9dad200-multus-conf-dir\") pod \"multus-4nqkj\" (UID: \"a2e3b25e-b6dc-46ef-86e8-4e70f9dad200\") " pod="openshift-multus/multus-4nqkj"
Oct 30 21:42:27.825620 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.824883    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a2e3b25e-b6dc-46ef-86e8-4e70f9dad200-host-var-lib-kubelet\") pod \"multus-4nqkj\" (UID: \"a2e3b25e-b6dc-46ef-86e8-4e70f9dad200\") " pod="openshift-multus/multus-4nqkj"
Oct 30 21:42:27.825620 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.824888    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/538d5103-1c92-48f1-acc7-f90956848727-run-ovn\") pod \"ovnkube-node-22ldv\" (UID: \"538d5103-1c92-48f1-acc7-f90956848727\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ldv"
Oct 30 21:42:27.825620 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.824900    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a2e3b25e-b6dc-46ef-86e8-4e70f9dad200-hostroot\") pod \"multus-4nqkj\" (UID: \"a2e3b25e-b6dc-46ef-86e8-4e70f9dad200\") " pod="openshift-multus/multus-4nqkj"
Oct 30 21:42:27.825620 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.824914    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a2e3b25e-b6dc-46ef-86e8-4e70f9dad200-host-var-lib-kubelet\") pod \"multus-4nqkj\" (UID: \"a2e3b25e-b6dc-46ef-86e8-4e70f9dad200\") " pod="openshift-multus/multus-4nqkj"
Oct 30 21:42:27.825620 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.824924    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a2e3b25e-b6dc-46ef-86e8-4e70f9dad200-hostroot\") pod \"multus-4nqkj\" (UID: \"a2e3b25e-b6dc-46ef-86e8-4e70f9dad200\") " pod="openshift-multus/multus-4nqkj"
Oct 30 21:42:27.825620 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.824918    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4427f59d-8003-440f-9646-a46b176cc4a7-socket-dir\") pod \"aws-ebs-csi-driver-node-qk9dg\" (UID: \"4427f59d-8003-440f-9646-a46b176cc4a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qk9dg"
Oct 30 21:42:27.825620 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.824888    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/538d5103-1c92-48f1-acc7-f90956848727-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-22ldv\" (UID: \"538d5103-1c92-48f1-acc7-f90956848727\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ldv"
Oct 30 21:42:27.825620 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.824948    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4427f59d-8003-440f-9646-a46b176cc4a7-socket-dir\") pod \"aws-ebs-csi-driver-node-qk9dg\" (UID: \"4427f59d-8003-440f-9646-a46b176cc4a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qk9dg"
Oct 30 21:42:27.825620 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.824954    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1f3b6a23-d7ea-4ce4-b1e8-68133c4b4ebc-cnibin\") pod \"multus-additional-cni-plugins-b47v2\" (UID: \"1f3b6a23-d7ea-4ce4-b1e8-68133c4b4ebc\") " pod="openshift-multus/multus-additional-cni-plugins-b47v2"
Oct 30 21:42:27.825620 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.824980    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cad67bc5-c8d2-497c-ba1b-cdbae5822cfa-registry-tls\") pod \"image-registry-b6cdbd45d-bdftd\" (UID: \"cad67bc5-c8d2-497c-ba1b-cdbae5822cfa\") " pod="openshift-image-registry/image-registry-b6cdbd45d-bdftd"
Oct 30 21:42:27.825620 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.824992    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1f3b6a23-d7ea-4ce4-b1e8-68133c4b4ebc-cnibin\") pod \"multus-additional-cni-plugins-b47v2\" (UID: \"1f3b6a23-d7ea-4ce4-b1e8-68133c4b4ebc\") " pod="openshift-multus/multus-additional-cni-plugins-b47v2"
Oct 30 21:42:27.825620 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.825013    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f648e933-f035-486a-88d5-1b185c89782e-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-j5xsr\" (UID: \"f648e933-f035-486a-88d5-1b185c89782e\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-j5xsr"
Oct 30 21:42:27.825620 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.825033    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ed08bc51-70e6-4658-adb3-ed36f5176552-etc-systemd\") pod \"tuned-hskxk\" (UID: \"ed08bc51-70e6-4658-adb3-ed36f5176552\") " pod="openshift-cluster-node-tuning-operator/tuned-hskxk"
Oct 30 21:42:27.825620 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.825047    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4427f59d-8003-440f-9646-a46b176cc4a7-registration-dir\") pod \"aws-ebs-csi-driver-node-qk9dg\" (UID: \"4427f59d-8003-440f-9646-a46b176cc4a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qk9dg"
Oct 30 21:42:27.825620 ip-10-0-130-60 kubenswrapper[5061]: E1030 21:42:27.825059    5061 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 21:42:27.825620 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.825064    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ed08bc51-70e6-4658-adb3-ed36f5176552-etc-sysctl-d\") pod \"tuned-hskxk\" (UID: \"ed08bc51-70e6-4658-adb3-ed36f5176552\") " pod="openshift-cluster-node-tuning-operator/tuned-hskxk"
Oct 30 21:42:27.825620 ip-10-0-130-60 kubenswrapper[5061]: E1030 21:42:27.825073    5061 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b6cdbd45d-bdftd: secret "image-registry-tls" not found
Oct 30 21:42:27.825620 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.825081    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/538d5103-1c92-48f1-acc7-f90956848727-run-systemd\") pod \"ovnkube-node-22ldv\" (UID: \"538d5103-1c92-48f1-acc7-f90956848727\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ldv"
Oct 30 21:42:27.825620 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.825102    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ed08bc51-70e6-4658-adb3-ed36f5176552-sys\") pod \"tuned-hskxk\" (UID: \"ed08bc51-70e6-4658-adb3-ed36f5176552\") " pod="openshift-cluster-node-tuning-operator/tuned-hskxk"
Oct 30 21:42:27.825620 ip-10-0-130-60 kubenswrapper[5061]: E1030 21:42:27.825116    5061 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cad67bc5-c8d2-497c-ba1b-cdbae5822cfa-registry-tls podName:cad67bc5-c8d2-497c-ba1b-cdbae5822cfa nodeName:}" failed. No retries permitted until 2026-10-30 21:42:28.32510281 +0000 UTC m=+1.937872559 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cad67bc5-c8d2-497c-ba1b-cdbae5822cfa-registry-tls") pod "image-registry-b6cdbd45d-bdftd" (UID: "cad67bc5-c8d2-497c-ba1b-cdbae5822cfa") : secret "image-registry-tls" not found
Oct 30 21:42:27.825620 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.825114    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ed08bc51-70e6-4658-adb3-ed36f5176552-etc-systemd\") pod \"tuned-hskxk\" (UID: \"ed08bc51-70e6-4658-adb3-ed36f5176552\") " pod="openshift-cluster-node-tuning-operator/tuned-hskxk"
Oct 30 21:42:27.825620 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.825137    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4427f59d-8003-440f-9646-a46b176cc4a7-registration-dir\") pod \"aws-ebs-csi-driver-node-qk9dg\" (UID: \"4427f59d-8003-440f-9646-a46b176cc4a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qk9dg"
Oct 30 21:42:27.825620 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.825148    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/538d5103-1c92-48f1-acc7-f90956848727-run-systemd\") pod \"ovnkube-node-22ldv\" (UID: \"538d5103-1c92-48f1-acc7-f90956848727\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ldv"
Oct 30 21:42:27.825620 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.825145    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d-metrics-certs\") pod \"router-default-6c7f96b45-4wj99\" (UID: \"9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d\") " pod="openshift-ingress/router-default-6c7f96b45-4wj99"
Oct 30 21:42:27.825620 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.825164    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ed08bc51-70e6-4658-adb3-ed36f5176552-sys\") pod \"tuned-hskxk\" (UID: \"ed08bc51-70e6-4658-adb3-ed36f5176552\") " pod="openshift-cluster-node-tuning-operator/tuned-hskxk"
Oct 30 21:42:27.825620 ip-10-0-130-60 kubenswrapper[5061]: E1030 21:42:27.825178    5061 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 21:42:27.825620 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.825191    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ed08bc51-70e6-4658-adb3-ed36f5176552-etc-kubernetes\") pod \"tuned-hskxk\" (UID: \"ed08bc51-70e6-4658-adb3-ed36f5176552\") " pod="openshift-cluster-node-tuning-operator/tuned-hskxk"
Oct 30 21:42:27.825620 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.825193    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ed08bc51-70e6-4658-adb3-ed36f5176552-etc-sysctl-d\") pod \"tuned-hskxk\" (UID: \"ed08bc51-70e6-4658-adb3-ed36f5176552\") " pod="openshift-cluster-node-tuning-operator/tuned-hskxk"
Oct 30 21:42:27.825620 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.825221    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0929b9f6-12ec-4f4d-8ba3-5cef9defba15-host-slash\") pod \"iptables-alerter-wbq4l\" (UID: \"0929b9f6-12ec-4f4d-8ba3-5cef9defba15\") " pod="openshift-network-operator/iptables-alerter-wbq4l"
Oct 30 21:42:27.825620 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.825228    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ed08bc51-70e6-4658-adb3-ed36f5176552-etc-kubernetes\") pod \"tuned-hskxk\" (UID: \"ed08bc51-70e6-4658-adb3-ed36f5176552\") " pod="openshift-cluster-node-tuning-operator/tuned-hskxk"
Oct 30 21:42:27.825620 ip-10-0-130-60 kubenswrapper[5061]: E1030 21:42:27.825250    5061 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f648e933-f035-486a-88d5-1b185c89782e-networking-console-plugin-cert podName:f648e933-f035-486a-88d5-1b185c89782e nodeName:}" failed. No retries permitted until 2026-10-30 21:42:28.325234827 +0000 UTC m=+1.938004581 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/f648e933-f035-486a-88d5-1b185c89782e-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-j5xsr" (UID: "f648e933-f035-486a-88d5-1b185c89782e") : secret "networking-console-plugin-cert" not found
Oct 30 21:42:27.825620 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.825267    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/84d9b682-3241-40af-87d5-f87d54ce3697-metrics-tls\") pod \"dns-default-6pttx\" (UID: \"84d9b682-3241-40af-87d5-f87d54ce3697\") " pod="openshift-dns/dns-default-6pttx"
Oct 30 21:42:27.825620 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.825287    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/74ff988a-c05e-4a3e-adff-5945b79cf248-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mstdj\" (UID: \"74ff988a-c05e-4a3e-adff-5945b79cf248\") " pod="openshift-insights/insights-runtime-extractor-mstdj"
Oct 30 21:42:27.825620 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.825265    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0929b9f6-12ec-4f4d-8ba3-5cef9defba15-host-slash\") pod \"iptables-alerter-wbq4l\" (UID: \"0929b9f6-12ec-4f4d-8ba3-5cef9defba15\") " pod="openshift-network-operator/iptables-alerter-wbq4l"
Oct 30 21:42:27.825620 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.825312    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0bb51175-fa5e-4472-8d83-106bca721585-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-tlwms\" (UID: \"0bb51175-fa5e-4472-8d83-106bca721585\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-tlwms"
Oct 30 21:42:27.825620 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.825337    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/538d5103-1c92-48f1-acc7-f90956848727-host-kubelet\") pod \"ovnkube-node-22ldv\" (UID: \"538d5103-1c92-48f1-acc7-f90956848727\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ldv"
Oct 30 21:42:27.825620 ip-10-0-130-60 kubenswrapper[5061]: E1030 21:42:27.825345    5061 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 21:42:27.825620 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.825362    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/538d5103-1c92-48f1-acc7-f90956848727-host-run-netns\") pod \"ovnkube-node-22ldv\" (UID: \"538d5103-1c92-48f1-acc7-f90956848727\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ldv"
Oct 30 21:42:27.825620 ip-10-0-130-60 kubenswrapper[5061]: E1030 21:42:27.825366    5061 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 21:42:27.825620 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.825387    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4427f59d-8003-440f-9646-a46b176cc4a7-sys-fs\") pod \"aws-ebs-csi-driver-node-qk9dg\" (UID: \"4427f59d-8003-440f-9646-a46b176cc4a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qk9dg"
Oct 30 21:42:27.825620 ip-10-0-130-60 kubenswrapper[5061]: E1030 21:42:27.825396    5061 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/74ff988a-c05e-4a3e-adff-5945b79cf248-kube-rbac-proxy-cm podName:74ff988a-c05e-4a3e-adff-5945b79cf248 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:28.325382914 +0000 UTC m=+1.938152667 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/74ff988a-c05e-4a3e-adff-5945b79cf248-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mstdj" (UID: "74ff988a-c05e-4a3e-adff-5945b79cf248") : configmap "kube-rbac-proxy" not found
Oct 30 21:42:27.825620 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.825415    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/538d5103-1c92-48f1-acc7-f90956848727-host-kubelet\") pod \"ovnkube-node-22ldv\" (UID: \"538d5103-1c92-48f1-acc7-f90956848727\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ldv"
Oct 30 21:42:27.825620 ip-10-0-130-60 kubenswrapper[5061]: E1030 21:42:27.825422    5061 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84d9b682-3241-40af-87d5-f87d54ce3697-metrics-tls podName:84d9b682-3241-40af-87d5-f87d54ce3697 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:28.325408548 +0000 UTC m=+1.938178297 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/84d9b682-3241-40af-87d5-f87d54ce3697-metrics-tls") pod "dns-default-6pttx" (UID: "84d9b682-3241-40af-87d5-f87d54ce3697") : secret "dns-default-metrics-tls" not found
Oct 30 21:42:27.825620 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.825432    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4427f59d-8003-440f-9646-a46b176cc4a7-sys-fs\") pod \"aws-ebs-csi-driver-node-qk9dg\" (UID: \"4427f59d-8003-440f-9646-a46b176cc4a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qk9dg"
Oct 30 21:42:27.825620 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.825424    5061 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 21:42:27.825620 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.825452    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/538d5103-1c92-48f1-acc7-f90956848727-etc-openvswitch\") pod \"ovnkube-node-22ldv\" (UID: \"538d5103-1c92-48f1-acc7-f90956848727\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ldv"
Oct 30 21:42:27.825620 ip-10-0-130-60 kubenswrapper[5061]: E1030 21:42:27.825460    5061 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 21:42:27.825620 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.825472    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/538d5103-1c92-48f1-acc7-f90956848727-etc-openvswitch\") pod \"ovnkube-node-22ldv\" (UID: \"538d5103-1c92-48f1-acc7-f90956848727\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ldv"
Oct 30 21:42:27.825620 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.825477    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/538d5103-1c92-48f1-acc7-f90956848727-log-socket\") pod \"ovnkube-node-22ldv\" (UID: \"538d5103-1c92-48f1-acc7-f90956848727\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ldv"
Oct 30 21:42:27.825620 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.825447    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/538d5103-1c92-48f1-acc7-f90956848727-host-run-netns\") pod \"ovnkube-node-22ldv\" (UID: \"538d5103-1c92-48f1-acc7-f90956848727\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ldv"
Oct 30 21:42:27.825620 ip-10-0-130-60 kubenswrapper[5061]: E1030 21:42:27.825544    5061 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0bb51175-fa5e-4472-8d83-106bca721585-samples-operator-tls podName:0bb51175-fa5e-4472-8d83-106bca721585 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:28.325512061 +0000 UTC m=+1.938281815 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0bb51175-fa5e-4472-8d83-106bca721585-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-tlwms" (UID: "0bb51175-fa5e-4472-8d83-106bca721585") : secret "samples-operator-tls" not found
Oct 30 21:42:27.825620 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.825563    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ed08bc51-70e6-4658-adb3-ed36f5176552-host\") pod \"tuned-hskxk\" (UID: \"ed08bc51-70e6-4658-adb3-ed36f5176552\") " pod="openshift-cluster-node-tuning-operator/tuned-hskxk"
Oct 30 21:42:27.825620 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.825581    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a2e3b25e-b6dc-46ef-86e8-4e70f9dad200-multus-cni-dir\") pod \"multus-4nqkj\" (UID: \"a2e3b25e-b6dc-46ef-86e8-4e70f9dad200\") " pod="openshift-multus/multus-4nqkj"
Oct 30 21:42:27.825620 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.825596    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a2e3b25e-b6dc-46ef-86e8-4e70f9dad200-host-run-multus-certs\") pod \"multus-4nqkj\" (UID: \"a2e3b25e-b6dc-46ef-86e8-4e70f9dad200\") " pod="openshift-multus/multus-4nqkj"
Oct 30 21:42:27.825620 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.825612    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/538d5103-1c92-48f1-acc7-f90956848727-var-lib-openvswitch\") pod \"ovnkube-node-22ldv\" (UID: \"538d5103-1c92-48f1-acc7-f90956848727\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ldv"
Oct 30 21:42:27.825620 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.825626    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/538d5103-1c92-48f1-acc7-f90956848727-node-log\") pod \"ovnkube-node-22ldv\" (UID: \"538d5103-1c92-48f1-acc7-f90956848727\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ldv"
Oct 30 21:42:27.825620 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.825640    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ed08bc51-70e6-4658-adb3-ed36f5176552-etc-sysconfig\") pod \"tuned-hskxk\" (UID: \"ed08bc51-70e6-4658-adb3-ed36f5176552\") " pod="openshift-cluster-node-tuning-operator/tuned-hskxk"
Oct 30 21:42:27.825620 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.825655    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ed08bc51-70e6-4658-adb3-ed36f5176552-run\") pod \"tuned-hskxk\" (UID: \"ed08bc51-70e6-4658-adb3-ed36f5176552\") " pod="openshift-cluster-node-tuning-operator/tuned-hskxk"
Oct 30 21:42:27.825620 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.825670    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/538d5103-1c92-48f1-acc7-f90956848727-host-cni-bin\") pod \"ovnkube-node-22ldv\" (UID: \"538d5103-1c92-48f1-acc7-f90956848727\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ldv"
Oct 30 21:42:27.827904 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.825684    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/613e7c8a-e2c7-424a-93e0-7fa86d18a8a9-registry-tls\") pod \"image-registry-d44c8cbb7-lmw97\" (UID: \"613e7c8a-e2c7-424a-93e0-7fa86d18a8a9\") " pod="openshift-image-registry/image-registry-d44c8cbb7-lmw97"
Oct 30 21:42:27.827904 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.825698    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ed08bc51-70e6-4658-adb3-ed36f5176552-var-lib-kubelet\") pod \"tuned-hskxk\" (UID: \"ed08bc51-70e6-4658-adb3-ed36f5176552\") " pod="openshift-cluster-node-tuning-operator/tuned-hskxk"
Oct 30 21:42:27.827904 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.825714    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a2e3b25e-b6dc-46ef-86e8-4e70f9dad200-host-var-lib-cni-bin\") pod \"multus-4nqkj\" (UID: \"a2e3b25e-b6dc-46ef-86e8-4e70f9dad200\") " pod="openshift-multus/multus-4nqkj"
Oct 30 21:42:27.827904 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.825729    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1f3b6a23-d7ea-4ce4-b1e8-68133c4b4ebc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-b47v2\" (UID: \"1f3b6a23-d7ea-4ce4-b1e8-68133c4b4ebc\") " pod="openshift-multus/multus-additional-cni-plugins-b47v2"
Oct 30 21:42:27.827904 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.825754    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4427f59d-8003-440f-9646-a46b176cc4a7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qk9dg\" (UID: \"4427f59d-8003-440f-9646-a46b176cc4a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qk9dg"
Oct 30 21:42:27.827904 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.825768    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/74ff988a-c05e-4a3e-adff-5945b79cf248-crio-socket\") pod \"insights-runtime-extractor-mstdj\" (UID: \"74ff988a-c05e-4a3e-adff-5945b79cf248\") " pod="openshift-insights/insights-runtime-extractor-mstdj"
Oct 30 21:42:27.827904 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.825786    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2205e8e6-f038-47fa-b393-0abe4d622e03-metrics-certs\") pod \"network-metrics-daemon-q6tjc\" (UID: \"2205e8e6-f038-47fa-b393-0abe4d622e03\") " pod="openshift-multus/network-metrics-daemon-q6tjc"
Oct 30 21:42:27.827904 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.825802    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a2e3b25e-b6dc-46ef-86e8-4e70f9dad200-host-run-k8s-cni-cncf-io\") pod \"multus-4nqkj\" (UID: \"a2e3b25e-b6dc-46ef-86e8-4e70f9dad200\") " pod="openshift-multus/multus-4nqkj"
Oct 30 21:42:27.827904 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.825818    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1f3b6a23-d7ea-4ce4-b1e8-68133c4b4ebc-system-cni-dir\") pod \"multus-additional-cni-plugins-b47v2\" (UID: \"1f3b6a23-d7ea-4ce4-b1e8-68133c4b4ebc\") " pod="openshift-multus/multus-additional-cni-plugins-b47v2"
Oct 30 21:42:27.827904 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.825883    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1f3b6a23-d7ea-4ce4-b1e8-68133c4b4ebc-system-cni-dir\") pod \"multus-additional-cni-plugins-b47v2\" (UID: \"1f3b6a23-d7ea-4ce4-b1e8-68133c4b4ebc\") " pod="openshift-multus/multus-additional-cni-plugins-b47v2"
Oct 30 21:42:27.827904 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.825930    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/538d5103-1c92-48f1-acc7-f90956848727-host-cni-bin\") pod \"ovnkube-node-22ldv\" (UID: \"538d5103-1c92-48f1-acc7-f90956848727\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ldv"
Oct 30 21:42:27.827904 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.825936    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ed08bc51-70e6-4658-adb3-ed36f5176552-run\") pod \"tuned-hskxk\" (UID: \"ed08bc51-70e6-4658-adb3-ed36f5176552\") " pod="openshift-cluster-node-tuning-operator/tuned-hskxk"
Oct 30 21:42:27.827904 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.825542    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/538d5103-1c92-48f1-acc7-f90956848727-log-socket\") pod \"ovnkube-node-22ldv\" (UID: \"538d5103-1c92-48f1-acc7-f90956848727\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ldv"
Oct 30 21:42:27.827904 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.825991    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ed08bc51-70e6-4658-adb3-ed36f5176552-host\") pod \"tuned-hskxk\" (UID: \"ed08bc51-70e6-4658-adb3-ed36f5176552\") " pod="openshift-cluster-node-tuning-operator/tuned-hskxk"
Oct 30 21:42:27.827904 ip-10-0-130-60 kubenswrapper[5061]: E1030 21:42:27.826001    5061 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 21:42:27.827904 ip-10-0-130-60 kubenswrapper[5061]: E1030 21:42:27.826013    5061 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-d44c8cbb7-lmw97: secret "image-registry-tls" not found
Oct 30 21:42:27.827904 ip-10-0-130-60 kubenswrapper[5061]: E1030 21:42:27.826050    5061 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/613e7c8a-e2c7-424a-93e0-7fa86d18a8a9-registry-tls podName:613e7c8a-e2c7-424a-93e0-7fa86d18a8a9 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:28.326036271 +0000 UTC m=+1.938806022 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/613e7c8a-e2c7-424a-93e0-7fa86d18a8a9-registry-tls") pod "image-registry-d44c8cbb7-lmw97" (UID: "613e7c8a-e2c7-424a-93e0-7fa86d18a8a9") : secret "image-registry-tls" not found
Oct 30 21:42:27.827904 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.826046    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a2e3b25e-b6dc-46ef-86e8-4e70f9dad200-multus-cni-dir\") pod \"multus-4nqkj\" (UID: \"a2e3b25e-b6dc-46ef-86e8-4e70f9dad200\") " pod="openshift-multus/multus-4nqkj"
Oct 30 21:42:27.827904 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.826082    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a2e3b25e-b6dc-46ef-86e8-4e70f9dad200-host-run-multus-certs\") pod \"multus-4nqkj\" (UID: \"a2e3b25e-b6dc-46ef-86e8-4e70f9dad200\") " pod="openshift-multus/multus-4nqkj"
Oct 30 21:42:27.827904 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.826097    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ed08bc51-70e6-4658-adb3-ed36f5176552-var-lib-kubelet\") pod \"tuned-hskxk\" (UID: \"ed08bc51-70e6-4658-adb3-ed36f5176552\") " pod="openshift-cluster-node-tuning-operator/tuned-hskxk"
Oct 30 21:42:27.827904 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.826132    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/538d5103-1c92-48f1-acc7-f90956848727-var-lib-openvswitch\") pod \"ovnkube-node-22ldv\" (UID: \"538d5103-1c92-48f1-acc7-f90956848727\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ldv"
Oct 30 21:42:27.827904 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.826153    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a2e3b25e-b6dc-46ef-86e8-4e70f9dad200-host-var-lib-cni-bin\") pod \"multus-4nqkj\" (UID: \"a2e3b25e-b6dc-46ef-86e8-4e70f9dad200\") " pod="openshift-multus/multus-4nqkj"
Oct 30 21:42:27.827904 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.826172    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/538d5103-1c92-48f1-acc7-f90956848727-node-log\") pod \"ovnkube-node-22ldv\" (UID: \"538d5103-1c92-48f1-acc7-f90956848727\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ldv"
Oct 30 21:42:27.827904 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.826206    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1f3b6a23-d7ea-4ce4-b1e8-68133c4b4ebc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-b47v2\" (UID: \"1f3b6a23-d7ea-4ce4-b1e8-68133c4b4ebc\") " pod="openshift-multus/multus-additional-cni-plugins-b47v2"
Oct 30 21:42:27.827904 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.826222    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ed08bc51-70e6-4658-adb3-ed36f5176552-etc-sysconfig\") pod \"tuned-hskxk\" (UID: \"ed08bc51-70e6-4658-adb3-ed36f5176552\") " pod="openshift-cluster-node-tuning-operator/tuned-hskxk"
Oct 30 21:42:27.827904 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.826254    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4427f59d-8003-440f-9646-a46b176cc4a7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qk9dg\" (UID: \"4427f59d-8003-440f-9646-a46b176cc4a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qk9dg"
Oct 30 21:42:27.827904 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.826302    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a2e3b25e-b6dc-46ef-86e8-4e70f9dad200-host-run-k8s-cni-cncf-io\") pod \"multus-4nqkj\" (UID: \"a2e3b25e-b6dc-46ef-86e8-4e70f9dad200\") " pod="openshift-multus/multus-4nqkj"
Oct 30 21:42:27.827904 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.826355    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/74ff988a-c05e-4a3e-adff-5945b79cf248-crio-socket\") pod \"insights-runtime-extractor-mstdj\" (UID: \"74ff988a-c05e-4a3e-adff-5945b79cf248\") " pod="openshift-insights/insights-runtime-extractor-mstdj"
Oct 30 21:42:27.827904 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.827710    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d-metrics-certs\") pod \"router-default-6c7f96b45-4wj99\" (UID: \"9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d\") " pod="openshift-ingress/router-default-6c7f96b45-4wj99"
Oct 30 21:42:27.828732 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.828437    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2205e8e6-f038-47fa-b393-0abe4d622e03-metrics-certs\") pod \"network-metrics-daemon-q6tjc\" (UID: \"2205e8e6-f038-47fa-b393-0abe4d622e03\") " pod="openshift-multus/network-metrics-daemon-q6tjc"
Oct 30 21:42:27.911467 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.911445    5061 scope.go:122] "RemoveContainer" containerID="fef0614d3ea43c5da061d6b88ea8aa29855cf27a2684dc399ca63400506200bb"
Oct 30 21:42:27.916288 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.916268    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-x8gnj_b227ad6a-9e59-415e-b7e6-f7c25e988abb/node-ca/0.log"
Oct 30 21:42:27.918404 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:27.918377    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q6tjc"
Oct 30 21:42:28.286204 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:28.286133    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-22ldv"
Oct 30 21:42:28.318120 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:28.318095    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-pzpgw_28210ad9-18f0-4ecf-9bfc-b256440f0ce8/serve-healthcheck-canary/0.log"
Oct 30 21:42:28.327303 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:28.327281    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-q6tjc"]
Oct 30 21:42:28.330204 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:28.330182    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/84d9b682-3241-40af-87d5-f87d54ce3697-metrics-tls\") pod \"dns-default-6pttx\" (UID: \"84d9b682-3241-40af-87d5-f87d54ce3697\") " pod="openshift-dns/dns-default-6pttx"
Oct 30 21:42:28.330250 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:28.330221    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/74ff988a-c05e-4a3e-adff-5945b79cf248-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mstdj\" (UID: \"74ff988a-c05e-4a3e-adff-5945b79cf248\") " pod="openshift-insights/insights-runtime-extractor-mstdj"
Oct 30 21:42:28.330337 ip-10-0-130-60 kubenswrapper[5061]: E1030 21:42:28.330319    5061 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 21:42:28.330370 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:28.330338    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0bb51175-fa5e-4472-8d83-106bca721585-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-tlwms\" (UID: \"0bb51175-fa5e-4472-8d83-106bca721585\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-tlwms"
Oct 30 21:42:28.330400 ip-10-0-130-60 kubenswrapper[5061]: E1030 21:42:28.330382    5061 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84d9b682-3241-40af-87d5-f87d54ce3697-metrics-tls podName:84d9b682-3241-40af-87d5-f87d54ce3697 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:29.330365814 +0000 UTC m=+2.943135567 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/84d9b682-3241-40af-87d5-f87d54ce3697-metrics-tls") pod "dns-default-6pttx" (UID: "84d9b682-3241-40af-87d5-f87d54ce3697") : secret "dns-default-metrics-tls" not found
Oct 30 21:42:28.330437 ip-10-0-130-60 kubenswrapper[5061]: E1030 21:42:28.330423    5061 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 21:42:28.330437 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:28.330426    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/613e7c8a-e2c7-424a-93e0-7fa86d18a8a9-registry-tls\") pod \"image-registry-d44c8cbb7-lmw97\" (UID: \"613e7c8a-e2c7-424a-93e0-7fa86d18a8a9\") " pod="openshift-image-registry/image-registry-d44c8cbb7-lmw97"
Oct 30 21:42:28.330550 ip-10-0-130-60 kubenswrapper[5061]: E1030 21:42:28.330436    5061 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 21:42:28.330550 ip-10-0-130-60 kubenswrapper[5061]: E1030 21:42:28.330461    5061 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0bb51175-fa5e-4472-8d83-106bca721585-samples-operator-tls podName:0bb51175-fa5e-4472-8d83-106bca721585 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:29.330449127 +0000 UTC m=+2.943218879 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0bb51175-fa5e-4472-8d83-106bca721585-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-tlwms" (UID: "0bb51175-fa5e-4472-8d83-106bca721585") : secret "samples-operator-tls" not found
Oct 30 21:42:28.330550 ip-10-0-130-60 kubenswrapper[5061]: E1030 21:42:28.330483    5061 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/74ff988a-c05e-4a3e-adff-5945b79cf248-kube-rbac-proxy-cm podName:74ff988a-c05e-4a3e-adff-5945b79cf248 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:29.3304685 +0000 UTC m=+2.943238241 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/74ff988a-c05e-4a3e-adff-5945b79cf248-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mstdj" (UID: "74ff988a-c05e-4a3e-adff-5945b79cf248") : configmap "kube-rbac-proxy" not found
Oct 30 21:42:28.330550 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:28.330522    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/74ff988a-c05e-4a3e-adff-5945b79cf248-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mstdj\" (UID: \"74ff988a-c05e-4a3e-adff-5945b79cf248\") " pod="openshift-insights/insights-runtime-extractor-mstdj"
Oct 30 21:42:28.330715 ip-10-0-130-60 kubenswrapper[5061]: E1030 21:42:28.330548    5061 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 21:42:28.330715 ip-10-0-130-60 kubenswrapper[5061]: E1030 21:42:28.330569    5061 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-d44c8cbb7-lmw97: secret "image-registry-tls" not found
Oct 30 21:42:28.330715 ip-10-0-130-60 kubenswrapper[5061]: E1030 21:42:28.330597    5061 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 21:42:28.330715 ip-10-0-130-60 kubenswrapper[5061]: E1030 21:42:28.330619    5061 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/613e7c8a-e2c7-424a-93e0-7fa86d18a8a9-registry-tls podName:613e7c8a-e2c7-424a-93e0-7fa86d18a8a9 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:29.330604485 +0000 UTC m=+2.943374239 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/613e7c8a-e2c7-424a-93e0-7fa86d18a8a9-registry-tls") pod "image-registry-d44c8cbb7-lmw97" (UID: "613e7c8a-e2c7-424a-93e0-7fa86d18a8a9") : secret "image-registry-tls" not found
Oct 30 21:42:28.330715 ip-10-0-130-60 kubenswrapper[5061]: E1030 21:42:28.330626    5061 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 21:42:28.330715 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:28.330566    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/102b8049-7349-4e2b-974a-44c901eebcf3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-ns759\" (UID: \"102b8049-7349-4e2b-974a-44c901eebcf3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ns759"
Oct 30 21:42:28.330715 ip-10-0-130-60 kubenswrapper[5061]: E1030 21:42:28.330639    5061 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74ff988a-c05e-4a3e-adff-5945b79cf248-insights-runtime-extractor-tls podName:74ff988a-c05e-4a3e-adff-5945b79cf248 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:29.330628213 +0000 UTC m=+2.943397977 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/74ff988a-c05e-4a3e-adff-5945b79cf248-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mstdj" (UID: "74ff988a-c05e-4a3e-adff-5945b79cf248") : secret "insights-runtime-extractor-tls" not found
Oct 30 21:42:28.330715 ip-10-0-130-60 kubenswrapper[5061]: E1030 21:42:28.330663    5061 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/102b8049-7349-4e2b-974a-44c901eebcf3-cluster-monitoring-operator-tls podName:102b8049-7349-4e2b-974a-44c901eebcf3 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:29.330649281 +0000 UTC m=+2.943419021 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/102b8049-7349-4e2b-974a-44c901eebcf3-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-ns759" (UID: "102b8049-7349-4e2b-974a-44c901eebcf3") : secret "cluster-monitoring-operator-tls" not found
Oct 30 21:42:28.330715 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:28.330696    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d-service-ca-bundle\") pod \"router-default-6c7f96b45-4wj99\" (UID: \"9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d\") " pod="openshift-ingress/router-default-6c7f96b45-4wj99"
Oct 30 21:42:28.331039 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:28.330769    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cad67bc5-c8d2-497c-ba1b-cdbae5822cfa-registry-tls\") pod \"image-registry-b6cdbd45d-bdftd\" (UID: \"cad67bc5-c8d2-497c-ba1b-cdbae5822cfa\") " pod="openshift-image-registry/image-registry-b6cdbd45d-bdftd"
Oct 30 21:42:28.331039 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:28.330803    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f648e933-f035-486a-88d5-1b185c89782e-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-j5xsr\" (UID: \"f648e933-f035-486a-88d5-1b185c89782e\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-j5xsr"
Oct 30 21:42:28.331039 ip-10-0-130-60 kubenswrapper[5061]: E1030 21:42:28.330818    5061 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 21:42:28.331039 ip-10-0-130-60 kubenswrapper[5061]: E1030 21:42:28.330827    5061 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b6cdbd45d-bdftd: secret "image-registry-tls" not found
Oct 30 21:42:28.331039 ip-10-0-130-60 kubenswrapper[5061]: E1030 21:42:28.330844    5061 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d-service-ca-bundle podName:9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d nodeName:}" failed. No retries permitted until 2026-10-30 21:42:29.330830906 +0000 UTC m=+2.943600658 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d-service-ca-bundle") pod "router-default-6c7f96b45-4wj99" (UID: "9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d") : configmap references non-existent config key: service-ca.crt
Oct 30 21:42:28.331039 ip-10-0-130-60 kubenswrapper[5061]: E1030 21:42:28.330865    5061 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cad67bc5-c8d2-497c-ba1b-cdbae5822cfa-registry-tls podName:cad67bc5-c8d2-497c-ba1b-cdbae5822cfa nodeName:}" failed. No retries permitted until 2026-10-30 21:42:29.330855804 +0000 UTC m=+2.943625567 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cad67bc5-c8d2-497c-ba1b-cdbae5822cfa-registry-tls") pod "image-registry-b6cdbd45d-bdftd" (UID: "cad67bc5-c8d2-497c-ba1b-cdbae5822cfa") : secret "image-registry-tls" not found
Oct 30 21:42:28.331039 ip-10-0-130-60 kubenswrapper[5061]: E1030 21:42:28.330899    5061 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 21:42:28.331039 ip-10-0-130-60 kubenswrapper[5061]: E1030 21:42:28.330926    5061 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f648e933-f035-486a-88d5-1b185c89782e-networking-console-plugin-cert podName:f648e933-f035-486a-88d5-1b185c89782e nodeName:}" failed. No retries permitted until 2026-10-30 21:42:29.330918468 +0000 UTC m=+2.943688255 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/f648e933-f035-486a-88d5-1b185c89782e-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-j5xsr" (UID: "f648e933-f035-486a-88d5-1b185c89782e") : secret "networking-console-plugin-cert" not found
Oct 30 21:42:28.361618 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:28.361590    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2205e8e6_f038_47fa_b393_0abe4d622e03.slice/crio-492b16220d64a15e83ebb16c8afc6eec77e1ca5631b77d8bb062364d4f0f41c5 WatchSource:0}: Error finding container 492b16220d64a15e83ebb16c8afc6eec77e1ca5631b77d8bb062364d4f0f41c5: Status 404 returned error can't find the container with id 492b16220d64a15e83ebb16c8afc6eec77e1ca5631b77d8bb062364d4f0f41c5
Oct 30 21:42:28.363713 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:28.363699    5061 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 21:42:28.517827 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:28.517800    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-q4khr_033356b5-3c05-4f49-ad6a-7c369259970d/migrator/0.log"
Oct 30 21:42:28.605043 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:28.604979    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-sclrt_5e74071e-0462-42b5-a05c-d90bb638478a/console-operator/2.log"
Oct 30 21:42:28.605416 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:28.605399    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-sclrt_5e74071e-0462-42b5-a05c-d90bb638478a/console-operator/1.log"
Oct 30 21:42:28.605496 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:28.605431    5061 generic.go:356] "Generic (PLEG): container finished" podID="5e74071e-0462-42b5-a05c-d90bb638478a" containerID="2cbd862a35e10e1c535ed71f5b3a03b15058a86cbfdf498bcc1216b9929f1d49" exitCode=255
Oct 30 21:42:28.605564 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:28.605519    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-sclrt" event={"ID":"5e74071e-0462-42b5-a05c-d90bb638478a","Type":"ContainerDied","Data":"2cbd862a35e10e1c535ed71f5b3a03b15058a86cbfdf498bcc1216b9929f1d49"}
Oct 30 21:42:28.605631 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:28.605603    5061 scope.go:122] "RemoveContainer" containerID="fef0614d3ea43c5da061d6b88ea8aa29855cf27a2684dc399ca63400506200bb"
Oct 30 21:42:28.606167 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:28.605904    5061 scope.go:122] "RemoveContainer" containerID="2cbd862a35e10e1c535ed71f5b3a03b15058a86cbfdf498bcc1216b9929f1d49"
Oct 30 21:42:28.606167 ip-10-0-130-60 kubenswrapper[5061]: E1030 21:42:28.606123    5061 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-sclrt_openshift-console-operator(5e74071e-0462-42b5-a05c-d90bb638478a)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-sclrt" podUID="5e74071e-0462-42b5-a05c-d90bb638478a"
Oct 30 21:42:28.606795 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:28.606770    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-q6tjc" event={"ID":"2205e8e6-f038-47fa-b393-0abe4d622e03","Type":"ContainerStarted","Data":"492b16220d64a15e83ebb16c8afc6eec77e1ca5631b77d8bb062364d4f0f41c5"}
Oct 30 21:42:28.717126 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:28.717100    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-rznzd_879022d4-06b9-44cc-a2a2-dc307b27345e/migrator/0.log"
Oct 30 21:42:28.797156 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:28.797122    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-979r9"
Oct 30 21:42:28.919650 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:28.919595    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-mgxcj_798f248f-d429-4f56-8752-e157dd94206b/kube-storage-version-migrator-operator/0.log"
Oct 30 21:42:29.339807 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:29.339769    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/74ff988a-c05e-4a3e-adff-5945b79cf248-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mstdj\" (UID: \"74ff988a-c05e-4a3e-adff-5945b79cf248\") " pod="openshift-insights/insights-runtime-extractor-mstdj"
Oct 30 21:42:29.339807 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:29.339810    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/102b8049-7349-4e2b-974a-44c901eebcf3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-ns759\" (UID: \"102b8049-7349-4e2b-974a-44c901eebcf3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ns759"
Oct 30 21:42:29.340030 ip-10-0-130-60 kubenswrapper[5061]: E1030 21:42:29.339912    5061 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 21:42:29.340030 ip-10-0-130-60 kubenswrapper[5061]: E1030 21:42:29.339917    5061 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 21:42:29.340030 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:29.339945    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d-service-ca-bundle\") pod \"router-default-6c7f96b45-4wj99\" (UID: \"9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d\") " pod="openshift-ingress/router-default-6c7f96b45-4wj99"
Oct 30 21:42:29.340030 ip-10-0-130-60 kubenswrapper[5061]: E1030 21:42:29.339966    5061 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/102b8049-7349-4e2b-974a-44c901eebcf3-cluster-monitoring-operator-tls podName:102b8049-7349-4e2b-974a-44c901eebcf3 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:31.339950298 +0000 UTC m=+4.952720058 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/102b8049-7349-4e2b-974a-44c901eebcf3-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-ns759" (UID: "102b8049-7349-4e2b-974a-44c901eebcf3") : secret "cluster-monitoring-operator-tls" not found
Oct 30 21:42:29.340030 ip-10-0-130-60 kubenswrapper[5061]: E1030 21:42:29.339987    5061 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74ff988a-c05e-4a3e-adff-5945b79cf248-insights-runtime-extractor-tls podName:74ff988a-c05e-4a3e-adff-5945b79cf248 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:31.339977538 +0000 UTC m=+4.952747277 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/74ff988a-c05e-4a3e-adff-5945b79cf248-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mstdj" (UID: "74ff988a-c05e-4a3e-adff-5945b79cf248") : secret "insights-runtime-extractor-tls" not found
Oct 30 21:42:29.340258 ip-10-0-130-60 kubenswrapper[5061]: E1030 21:42:29.340047    5061 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d-service-ca-bundle podName:9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d nodeName:}" failed. No retries permitted until 2026-10-30 21:42:31.340032301 +0000 UTC m=+4.952802051 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d-service-ca-bundle") pod "router-default-6c7f96b45-4wj99" (UID: "9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d") : configmap references non-existent config key: service-ca.crt
Oct 30 21:42:29.340258 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:29.340091    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cad67bc5-c8d2-497c-ba1b-cdbae5822cfa-registry-tls\") pod \"image-registry-b6cdbd45d-bdftd\" (UID: \"cad67bc5-c8d2-497c-ba1b-cdbae5822cfa\") " pod="openshift-image-registry/image-registry-b6cdbd45d-bdftd"
Oct 30 21:42:29.340258 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:29.340120    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f648e933-f035-486a-88d5-1b185c89782e-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-j5xsr\" (UID: \"f648e933-f035-486a-88d5-1b185c89782e\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-j5xsr"
Oct 30 21:42:29.340258 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:29.340167    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/84d9b682-3241-40af-87d5-f87d54ce3697-metrics-tls\") pod \"dns-default-6pttx\" (UID: \"84d9b682-3241-40af-87d5-f87d54ce3697\") " pod="openshift-dns/dns-default-6pttx"
Oct 30 21:42:29.340258 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:29.340191    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/74ff988a-c05e-4a3e-adff-5945b79cf248-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mstdj\" (UID: \"74ff988a-c05e-4a3e-adff-5945b79cf248\") " pod="openshift-insights/insights-runtime-extractor-mstdj"
Oct 30 21:42:29.340258 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:29.340215    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0bb51175-fa5e-4472-8d83-106bca721585-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-tlwms\" (UID: \"0bb51175-fa5e-4472-8d83-106bca721585\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-tlwms"
Oct 30 21:42:29.340258 ip-10-0-130-60 kubenswrapper[5061]: E1030 21:42:29.340229    5061 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 21:42:29.340611 ip-10-0-130-60 kubenswrapper[5061]: E1030 21:42:29.340270    5061 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f648e933-f035-486a-88d5-1b185c89782e-networking-console-plugin-cert podName:f648e933-f035-486a-88d5-1b185c89782e nodeName:}" failed. No retries permitted until 2026-10-30 21:42:31.340258899 +0000 UTC m=+4.953028638 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/f648e933-f035-486a-88d5-1b185c89782e-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-j5xsr" (UID: "f648e933-f035-486a-88d5-1b185c89782e") : secret "networking-console-plugin-cert" not found
Oct 30 21:42:29.340611 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:29.340291    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/613e7c8a-e2c7-424a-93e0-7fa86d18a8a9-registry-tls\") pod \"image-registry-d44c8cbb7-lmw97\" (UID: \"613e7c8a-e2c7-424a-93e0-7fa86d18a8a9\") " pod="openshift-image-registry/image-registry-d44c8cbb7-lmw97"
Oct 30 21:42:29.340611 ip-10-0-130-60 kubenswrapper[5061]: E1030 21:42:29.340306    5061 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 21:42:29.340611 ip-10-0-130-60 kubenswrapper[5061]: E1030 21:42:29.340320    5061 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 21:42:29.340611 ip-10-0-130-60 kubenswrapper[5061]: E1030 21:42:29.340357    5061 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0bb51175-fa5e-4472-8d83-106bca721585-samples-operator-tls podName:0bb51175-fa5e-4472-8d83-106bca721585 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:31.340346853 +0000 UTC m=+4.953116589 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0bb51175-fa5e-4472-8d83-106bca721585-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-tlwms" (UID: "0bb51175-fa5e-4472-8d83-106bca721585") : secret "samples-operator-tls" not found
Oct 30 21:42:29.340611 ip-10-0-130-60 kubenswrapper[5061]: E1030 21:42:29.340370    5061 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84d9b682-3241-40af-87d5-f87d54ce3697-metrics-tls podName:84d9b682-3241-40af-87d5-f87d54ce3697 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:31.340364466 +0000 UTC m=+4.953134202 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/84d9b682-3241-40af-87d5-f87d54ce3697-metrics-tls") pod "dns-default-6pttx" (UID: "84d9b682-3241-40af-87d5-f87d54ce3697") : secret "dns-default-metrics-tls" not found
Oct 30 21:42:29.340997 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:29.340752    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/74ff988a-c05e-4a3e-adff-5945b79cf248-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mstdj\" (UID: \"74ff988a-c05e-4a3e-adff-5945b79cf248\") " pod="openshift-insights/insights-runtime-extractor-mstdj"
Oct 30 21:42:29.342581 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:29.342559    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/613e7c8a-e2c7-424a-93e0-7fa86d18a8a9-registry-tls\") pod \"image-registry-d44c8cbb7-lmw97\" (UID: \"613e7c8a-e2c7-424a-93e0-7fa86d18a8a9\") " pod="openshift-image-registry/image-registry-d44c8cbb7-lmw97"
Oct 30 21:42:29.342581 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:29.342575    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cad67bc5-c8d2-497c-ba1b-cdbae5822cfa-registry-tls\") pod \"image-registry-b6cdbd45d-bdftd\" (UID: \"cad67bc5-c8d2-497c-ba1b-cdbae5822cfa\") " pod="openshift-image-registry/image-registry-b6cdbd45d-bdftd"
Oct 30 21:42:29.411792 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:29.411772    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-d44c8cbb7-lmw97"
Oct 30 21:42:29.423120 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:29.423096    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-b6cdbd45d-bdftd"
Oct 30 21:42:29.612904 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:29.612827    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-sclrt_5e74071e-0462-42b5-a05c-d90bb638478a/console-operator/2.log"
Oct 30 21:42:29.895599 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:29.895574    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-d44c8cbb7-lmw97"]
Oct 30 21:42:29.898222 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:29.898194    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod613e7c8a_e2c7_424a_93e0_7fa86d18a8a9.slice/crio-cf44e7310f7a985eddb19ab993be5a1e859fbae93f90c65ec347d84f93c73eae WatchSource:0}: Error finding container cf44e7310f7a985eddb19ab993be5a1e859fbae93f90c65ec347d84f93c73eae: Status 404 returned error can't find the container with id cf44e7310f7a985eddb19ab993be5a1e859fbae93f90c65ec347d84f93c73eae
Oct 30 21:42:29.900297 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:29.900279    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-b6cdbd45d-bdftd"]
Oct 30 21:42:29.946666 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:29.946637    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcad67bc5_c8d2_497c_ba1b_cdbae5822cfa.slice/crio-5fc41a9f51634495988a3328fa35ed11319cfa44aa6fce95ebb4811cdb77cddd WatchSource:0}: Error finding container 5fc41a9f51634495988a3328fa35ed11319cfa44aa6fce95ebb4811cdb77cddd: Status 404 returned error can't find the container with id 5fc41a9f51634495988a3328fa35ed11319cfa44aa6fce95ebb4811cdb77cddd
Oct 30 21:42:30.376958 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:30.376935    5061 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-sclrt"
Oct 30 21:42:30.377312 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:30.377295    5061 scope.go:122] "RemoveContainer" containerID="2cbd862a35e10e1c535ed71f5b3a03b15058a86cbfdf498bcc1216b9929f1d49"
Oct 30 21:42:30.377492 ip-10-0-130-60 kubenswrapper[5061]: E1030 21:42:30.377474    5061 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-sclrt_openshift-console-operator(5e74071e-0462-42b5-a05c-d90bb638478a)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-sclrt" podUID="5e74071e-0462-42b5-a05c-d90bb638478a"
Oct 30 21:42:30.617391 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:30.617358    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-b6cdbd45d-bdftd" event={"ID":"cad67bc5-c8d2-497c-ba1b-cdbae5822cfa","Type":"ContainerStarted","Data":"425394fcf72ef30f2fc6caca30666a86ae8e21732166003e89071dc2e5089383"}
Oct 30 21:42:30.617876 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:30.617412    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-b6cdbd45d-bdftd" event={"ID":"cad67bc5-c8d2-497c-ba1b-cdbae5822cfa","Type":"ContainerStarted","Data":"5fc41a9f51634495988a3328fa35ed11319cfa44aa6fce95ebb4811cdb77cddd"}
Oct 30 21:42:30.619178 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:30.619151    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-q6tjc" event={"ID":"2205e8e6-f038-47fa-b393-0abe4d622e03","Type":"ContainerStarted","Data":"4fa7c726b3182daa704b357320aaf2cac339b90c2edceba06bf8c47e1ab09ca2"}
Oct 30 21:42:30.619277 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:30.619199    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-q6tjc" event={"ID":"2205e8e6-f038-47fa-b393-0abe4d622e03","Type":"ContainerStarted","Data":"e7058e94a5c48626ab7eca6d19becba631b0ff940f120393e8da0ff44b68e559"}
Oct 30 21:42:30.620669 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:30.620644    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-d44c8cbb7-lmw97" event={"ID":"613e7c8a-e2c7-424a-93e0-7fa86d18a8a9","Type":"ContainerStarted","Data":"0aaa71c56c29f494a17f99a4237e93c1f698b737a3e83a9a3a0aca110714021d"}
Oct 30 21:42:30.620748 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:30.620687    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-d44c8cbb7-lmw97" event={"ID":"613e7c8a-e2c7-424a-93e0-7fa86d18a8a9","Type":"ContainerStarted","Data":"cf44e7310f7a985eddb19ab993be5a1e859fbae93f90c65ec347d84f93c73eae"}
Oct 30 21:42:30.620748 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:30.620710    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-d44c8cbb7-lmw97"
Oct 30 21:42:31.360517 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:31.360478    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/84d9b682-3241-40af-87d5-f87d54ce3697-metrics-tls\") pod \"dns-default-6pttx\" (UID: \"84d9b682-3241-40af-87d5-f87d54ce3697\") " pod="openshift-dns/dns-default-6pttx"
Oct 30 21:42:31.360763 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:31.360523    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0bb51175-fa5e-4472-8d83-106bca721585-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-tlwms\" (UID: \"0bb51175-fa5e-4472-8d83-106bca721585\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-tlwms"
Oct 30 21:42:31.360763 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:31.360603    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/74ff988a-c05e-4a3e-adff-5945b79cf248-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mstdj\" (UID: \"74ff988a-c05e-4a3e-adff-5945b79cf248\") " pod="openshift-insights/insights-runtime-extractor-mstdj"
Oct 30 21:42:31.360763 ip-10-0-130-60 kubenswrapper[5061]: E1030 21:42:31.360623    5061 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 21:42:31.360763 ip-10-0-130-60 kubenswrapper[5061]: E1030 21:42:31.360691    5061 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 21:42:31.360763 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:31.360632    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/102b8049-7349-4e2b-974a-44c901eebcf3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-ns759\" (UID: \"102b8049-7349-4e2b-974a-44c901eebcf3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ns759"
Oct 30 21:42:31.360763 ip-10-0-130-60 kubenswrapper[5061]: E1030 21:42:31.360695    5061 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84d9b682-3241-40af-87d5-f87d54ce3697-metrics-tls podName:84d9b682-3241-40af-87d5-f87d54ce3697 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:35.360679404 +0000 UTC m=+8.973449140 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/84d9b682-3241-40af-87d5-f87d54ce3697-metrics-tls") pod "dns-default-6pttx" (UID: "84d9b682-3241-40af-87d5-f87d54ce3697") : secret "dns-default-metrics-tls" not found
Oct 30 21:42:31.360763 ip-10-0-130-60 kubenswrapper[5061]: E1030 21:42:31.360692    5061 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 21:42:31.361027 ip-10-0-130-60 kubenswrapper[5061]: E1030 21:42:31.360697    5061 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 21:42:31.361027 ip-10-0-130-60 kubenswrapper[5061]: E1030 21:42:31.360772    5061 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0bb51175-fa5e-4472-8d83-106bca721585-samples-operator-tls podName:0bb51175-fa5e-4472-8d83-106bca721585 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:35.360759834 +0000 UTC m=+8.973529570 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0bb51175-fa5e-4472-8d83-106bca721585-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-tlwms" (UID: "0bb51175-fa5e-4472-8d83-106bca721585") : secret "samples-operator-tls" not found
Oct 30 21:42:31.361027 ip-10-0-130-60 kubenswrapper[5061]: E1030 21:42:31.360828    5061 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74ff988a-c05e-4a3e-adff-5945b79cf248-insights-runtime-extractor-tls podName:74ff988a-c05e-4a3e-adff-5945b79cf248 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:35.360817187 +0000 UTC m=+8.973586923 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/74ff988a-c05e-4a3e-adff-5945b79cf248-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mstdj" (UID: "74ff988a-c05e-4a3e-adff-5945b79cf248") : secret "insights-runtime-extractor-tls" not found
Oct 30 21:42:31.361027 ip-10-0-130-60 kubenswrapper[5061]: E1030 21:42:31.360841    5061 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/102b8049-7349-4e2b-974a-44c901eebcf3-cluster-monitoring-operator-tls podName:102b8049-7349-4e2b-974a-44c901eebcf3 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:35.360834971 +0000 UTC m=+8.973604707 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/102b8049-7349-4e2b-974a-44c901eebcf3-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-ns759" (UID: "102b8049-7349-4e2b-974a-44c901eebcf3") : secret "cluster-monitoring-operator-tls" not found
Oct 30 21:42:31.361027 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:31.360869    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d-service-ca-bundle\") pod \"router-default-6c7f96b45-4wj99\" (UID: \"9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d\") " pod="openshift-ingress/router-default-6c7f96b45-4wj99"
Oct 30 21:42:31.361027 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:31.360920    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f648e933-f035-486a-88d5-1b185c89782e-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-j5xsr\" (UID: \"f648e933-f035-486a-88d5-1b185c89782e\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-j5xsr"
Oct 30 21:42:31.361459 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:31.361441    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d-service-ca-bundle\") pod \"router-default-6c7f96b45-4wj99\" (UID: \"9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d\") " pod="openshift-ingress/router-default-6c7f96b45-4wj99"
Oct 30 21:42:31.363286 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:31.363257    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f648e933-f035-486a-88d5-1b185c89782e-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-j5xsr\" (UID: \"f648e933-f035-486a-88d5-1b185c89782e\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-j5xsr"
Oct 30 21:42:31.511977 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:31.511945    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6c7f96b45-4wj99"
Oct 30 21:42:31.517189 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:31.517164    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-j5xsr"
Oct 30 21:42:31.628274 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:31.627095    5061 generic.go:356] "Generic (PLEG): container finished" podID="798f248f-d429-4f56-8752-e157dd94206b" containerID="c85648ba4fc4a16b4488b01235e3fa82e7c4592cc97d60f71a6e82d135601002" exitCode=0
Oct 30 21:42:31.628274 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:31.627177    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-mgxcj" event={"ID":"798f248f-d429-4f56-8752-e157dd94206b","Type":"ContainerDied","Data":"c85648ba4fc4a16b4488b01235e3fa82e7c4592cc97d60f71a6e82d135601002"}
Oct 30 21:42:31.628274 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:31.627220    5061 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 21:42:31.628274 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:31.627953    5061 scope.go:122] "RemoveContainer" containerID="c85648ba4fc4a16b4488b01235e3fa82e7c4592cc97d60f71a6e82d135601002"
Oct 30 21:42:31.637847 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:31.637825    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-22ldv"
Oct 30 21:42:31.638023 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:31.638010    5061 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 21:42:31.832027 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:31.832001    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-sztmx"
Oct 30 21:42:31.943419 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:31.943356    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-j5xsr"]
Oct 30 21:42:31.947608 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:31.947559    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf648e933_f035_486a_88d5_1b185c89782e.slice/crio-aa1c97ab65e7b16e37f6ab059cb1ca5a7b9a42c9280f35f147faa2ee1c43be9a WatchSource:0}: Error finding container aa1c97ab65e7b16e37f6ab059cb1ca5a7b9a42c9280f35f147faa2ee1c43be9a: Status 404 returned error can't find the container with id aa1c97ab65e7b16e37f6ab059cb1ca5a7b9a42c9280f35f147faa2ee1c43be9a
Oct 30 21:42:31.948073 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:31.948053    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6c7f96b45-4wj99"]
Oct 30 21:42:31.949513 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:31.949486    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f96d744_56f0_4df2_a9a5_2a6bda6d8d0d.slice/crio-d96248079e78dc311cf0f206d9483d107c2a5d1e6cdf69b3970bb29f3d9e71d3 WatchSource:0}: Error finding container d96248079e78dc311cf0f206d9483d107c2a5d1e6cdf69b3970bb29f3d9e71d3: Status 404 returned error can't find the container with id d96248079e78dc311cf0f206d9483d107c2a5d1e6cdf69b3970bb29f3d9e71d3
Oct 30 21:42:32.632924 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:32.632889    5061 generic.go:356] "Generic (PLEG): container finished" podID="0126a0a0-a1c8-4db3-a6af-629360e247f4" containerID="cd90312306fd8b8a34ad012c7004d017560ef3e5826d1d1371868cc75a772dee" exitCode=0
Oct 30 21:42:32.633336 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:32.632964    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vqwlt" event={"ID":"0126a0a0-a1c8-4db3-a6af-629360e247f4","Type":"ContainerDied","Data":"cd90312306fd8b8a34ad012c7004d017560ef3e5826d1d1371868cc75a772dee"}
Oct 30 21:42:32.633397 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:32.633345    5061 scope.go:122] "RemoveContainer" containerID="cd90312306fd8b8a34ad012c7004d017560ef3e5826d1d1371868cc75a772dee"
Oct 30 21:42:32.634992 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:32.634967    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-mgxcj" event={"ID":"798f248f-d429-4f56-8752-e157dd94206b","Type":"ContainerStarted","Data":"b8e1f4123d0f0888f8baeb8f7dd8faa6e7e9fbfd76255922aeae07f922131db4"}
Oct 30 21:42:32.636190 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:32.636172    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-j5xsr" event={"ID":"f648e933-f035-486a-88d5-1b185c89782e","Type":"ContainerStarted","Data":"aa1c97ab65e7b16e37f6ab059cb1ca5a7b9a42c9280f35f147faa2ee1c43be9a"}
Oct 30 21:42:32.637471 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:32.637449    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6c7f96b45-4wj99" event={"ID":"9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d","Type":"ContainerStarted","Data":"ece130026a498e57eccc79acaddf7169f9553736bb99c632165ae20f37b88641"}
Oct 30 21:42:32.637574 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:32.637484    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6c7f96b45-4wj99" event={"ID":"9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d","Type":"ContainerStarted","Data":"d96248079e78dc311cf0f206d9483d107c2a5d1e6cdf69b3970bb29f3d9e71d3"}
Oct 30 21:42:33.297736 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:33.297700    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-sclrt"
Oct 30 21:42:33.298147 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:33.298124    5061 scope.go:122] "RemoveContainer" containerID="2cbd862a35e10e1c535ed71f5b3a03b15058a86cbfdf498bcc1216b9929f1d49"
Oct 30 21:42:33.298337 ip-10-0-130-60 kubenswrapper[5061]: E1030 21:42:33.298318    5061 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-sclrt_openshift-console-operator(5e74071e-0462-42b5-a05c-d90bb638478a)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-sclrt" podUID="5e74071e-0462-42b5-a05c-d90bb638478a"
Oct 30 21:42:33.512268 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:33.512230    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-6c7f96b45-4wj99"
Oct 30 21:42:33.515279 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:33.515252    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-6c7f96b45-4wj99"
Oct 30 21:42:33.643323 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:33.643250    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vqwlt" event={"ID":"0126a0a0-a1c8-4db3-a6af-629360e247f4","Type":"ContainerStarted","Data":"f0a774f6f52b36c763837a8b0dfea5b46059636e4396b68f7f18bb4f6e560a63"}
Oct 30 21:42:33.796315 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:33.796286    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-b6cdbd45d-bdftd"
Oct 30 21:42:34.648769 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:34.648738    5061 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 21:42:34.649172 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:34.648737    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-j5xsr" event={"ID":"f648e933-f035-486a-88d5-1b185c89782e","Type":"ContainerStarted","Data":"0fb2849e99bc7938e10d8c28f7f75c7894c1b26fcf65eb03a2e6bfefe98377f2"}
Oct 30 21:42:35.396775 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:35.396739    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/84d9b682-3241-40af-87d5-f87d54ce3697-metrics-tls\") pod \"dns-default-6pttx\" (UID: \"84d9b682-3241-40af-87d5-f87d54ce3697\") " pod="openshift-dns/dns-default-6pttx"
Oct 30 21:42:35.396775 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:35.396778    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0bb51175-fa5e-4472-8d83-106bca721585-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-tlwms\" (UID: \"0bb51175-fa5e-4472-8d83-106bca721585\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-tlwms"
Oct 30 21:42:35.396967 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:35.396817    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/74ff988a-c05e-4a3e-adff-5945b79cf248-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mstdj\" (UID: \"74ff988a-c05e-4a3e-adff-5945b79cf248\") " pod="openshift-insights/insights-runtime-extractor-mstdj"
Oct 30 21:42:35.396967 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:35.396834    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/102b8049-7349-4e2b-974a-44c901eebcf3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-ns759\" (UID: \"102b8049-7349-4e2b-974a-44c901eebcf3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ns759"
Oct 30 21:42:35.396967 ip-10-0-130-60 kubenswrapper[5061]: E1030 21:42:35.396935    5061 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 21:42:35.397063 ip-10-0-130-60 kubenswrapper[5061]: E1030 21:42:35.396993    5061 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74ff988a-c05e-4a3e-adff-5945b79cf248-insights-runtime-extractor-tls podName:74ff988a-c05e-4a3e-adff-5945b79cf248 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:43.396980199 +0000 UTC m=+17.009749935 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/74ff988a-c05e-4a3e-adff-5945b79cf248-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mstdj" (UID: "74ff988a-c05e-4a3e-adff-5945b79cf248") : secret "insights-runtime-extractor-tls" not found
Oct 30 21:42:35.399265 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:35.399238    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/84d9b682-3241-40af-87d5-f87d54ce3697-metrics-tls\") pod \"dns-default-6pttx\" (UID: \"84d9b682-3241-40af-87d5-f87d54ce3697\") " pod="openshift-dns/dns-default-6pttx"
Oct 30 21:42:35.399265 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:35.399252    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/102b8049-7349-4e2b-974a-44c901eebcf3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-ns759\" (UID: \"102b8049-7349-4e2b-974a-44c901eebcf3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ns759"
Oct 30 21:42:35.399403 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:35.399288    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0bb51175-fa5e-4472-8d83-106bca721585-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-tlwms\" (UID: \"0bb51175-fa5e-4472-8d83-106bca721585\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-tlwms"
Oct 30 21:42:35.411153 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:35.411134    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ns759"
Oct 30 21:42:35.413281 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:35.413264    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-tlwms"
Oct 30 21:42:35.424442 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:35.424425    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6pttx"
Oct 30 21:42:35.854562 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:35.854519    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-ns759"]
Oct 30 21:42:35.855710 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:35.855690    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-tlwms"]
Oct 30 21:42:35.890934 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:35.890901    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod102b8049_7349_4e2b_974a_44c901eebcf3.slice/crio-cb4a9d70a32ce7fe1038de764286a965b1a0772068b96ecf115f56433360ec33 WatchSource:0}: Error finding container cb4a9d70a32ce7fe1038de764286a965b1a0772068b96ecf115f56433360ec33: Status 404 returned error can't find the container with id cb4a9d70a32ce7fe1038de764286a965b1a0772068b96ecf115f56433360ec33
Oct 30 21:42:35.901045 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:35.901028    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-6pttx"]
Oct 30 21:42:35.904462 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:35.904442    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84d9b682_3241_40af_87d5_f87d54ce3697.slice/crio-858560b92a066a0bb6155737d96dda864e7b072ed6b96ab66f7cc59f8511b140 WatchSource:0}: Error finding container 858560b92a066a0bb6155737d96dda864e7b072ed6b96ab66f7cc59f8511b140: Status 404 returned error can't find the container with id 858560b92a066a0bb6155737d96dda864e7b072ed6b96ab66f7cc59f8511b140
Oct 30 21:42:36.658251 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:36.658212    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ns759" event={"ID":"102b8049-7349-4e2b-974a-44c901eebcf3","Type":"ContainerStarted","Data":"cb4a9d70a32ce7fe1038de764286a965b1a0772068b96ecf115f56433360ec33"}
Oct 30 21:42:36.661886 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:36.661854    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6pttx" event={"ID":"84d9b682-3241-40af-87d5-f87d54ce3697","Type":"ContainerStarted","Data":"858560b92a066a0bb6155737d96dda864e7b072ed6b96ab66f7cc59f8511b140"}
Oct 30 21:42:36.663158 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:36.663122    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-tlwms" event={"ID":"0bb51175-fa5e-4472-8d83-106bca721585","Type":"ContainerStarted","Data":"7e5f30dad74c67fa503becafb74d034cda733ab2a94851bd1da9b7ce1d5af02b"}
Oct 30 21:42:39.521461 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:39.521435    5061 scope.go:122] "RemoveContainer" containerID="2cbd862a35e10e1c535ed71f5b3a03b15058a86cbfdf498bcc1216b9929f1d49"
Oct 30 21:42:39.679122 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:39.679085    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6pttx" event={"ID":"84d9b682-3241-40af-87d5-f87d54ce3697","Type":"ContainerStarted","Data":"63a4be9c9e76630996cb30cae1dda0cc4d1c36f0eed5c45aade985130bbfa430"}
Oct 30 21:42:39.679290 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:39.679144    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6pttx" event={"ID":"84d9b682-3241-40af-87d5-f87d54ce3697","Type":"ContainerStarted","Data":"bdd603b0d3c200fe02cd886cf1cf6ca96d9f73e2f46e70e932f2ef152cc03c7e"}
Oct 30 21:42:39.679290 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:39.679179    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-6pttx"
Oct 30 21:42:39.680922 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:39.680884    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-tlwms" event={"ID":"0bb51175-fa5e-4472-8d83-106bca721585","Type":"ContainerStarted","Data":"d3f3e588b09cdeda0e244e19236f06ec0bf8eefe0d42ca00245a8fdd062d7208"}
Oct 30 21:42:39.681043 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:39.680939    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-tlwms" event={"ID":"0bb51175-fa5e-4472-8d83-106bca721585","Type":"ContainerStarted","Data":"fff2fa53c4b0a28abed4125ddfb850359cf5088aa213b27e322f3b5fdaeb082c"}
Oct 30 21:42:39.682707 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:39.682687    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-sclrt_5e74071e-0462-42b5-a05c-d90bb638478a/console-operator/2.log"
Oct 30 21:42:39.682795 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:39.682765    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-sclrt" event={"ID":"5e74071e-0462-42b5-a05c-d90bb638478a","Type":"ContainerStarted","Data":"c85c96a96a7724b9cc87eebf713c62ed28acd4c31f9be5891df40b10cd90e9df"}
Oct 30 21:42:39.683459 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:39.683436    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-sclrt"
Oct 30 21:42:39.684192 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:39.684161    5061 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-sclrt container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.17:8443/readyz\": dial tcp 10.132.0.17:8443: connect: connection refused" start-of-body=
Oct 30 21:42:39.684278 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:39.684210    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-sclrt" podUID="5e74071e-0462-42b5-a05c-d90bb638478a" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.17:8443/readyz\": dial tcp 10.132.0.17:8443: connect: connection refused"
Oct 30 21:42:39.684681 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:39.684665    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ns759" event={"ID":"102b8049-7349-4e2b-974a-44c901eebcf3","Type":"ContainerStarted","Data":"190aca668c2174155b0b32a8be18a048f701cbda665d600c1b65d9067061cd78"}
Oct 30 21:42:39.773520 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:39.773488    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-6c7f96b45-4wj99"
Oct 30 21:42:39.773741 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:39.773632    5061 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 21:42:39.774761 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:39.774740    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-6c7f96b45-4wj99"
Oct 30 21:42:40.694593 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:40.694561    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-sclrt"
Oct 30 21:42:42.076583 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:42.076554    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-4s89b"]
Oct 30 21:42:42.098159 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:42.098136    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-4s89b"
Oct 30 21:42:42.110637 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:42.110604    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-8pk6d"]
Oct 30 21:42:42.122748 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:42.122724    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-8pk6d"
Oct 30 21:42:42.124191 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:42.124166    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-d44c8cbb7-lmw97"]
Oct 30 21:42:42.148759 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:42.148734    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Oct 30 21:42:42.157305 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:42.157287    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Oct 30 21:42:42.157396 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:42.157340    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-t55nj\""
Oct 30 21:42:42.222417 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:42.222382    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5d4d6cb75d-hrc4d"]
Oct 30 21:42:42.226387 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:42.226357    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d4d6cb75d-hrc4d"
Oct 30 21:42:42.249258 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:42.249220    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-7jzwb\""
Oct 30 21:42:42.249375 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:42.249363    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Oct 30 21:42:42.255999 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:42.255969    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-8pk6d"]
Oct 30 21:42:42.257563 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:42.257522    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/6339c2e5-a2c9-4e70-88e6-97285a01a8ff-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-8pk6d\" (UID: \"6339c2e5-a2c9-4e70-88e6-97285a01a8ff\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-8pk6d"
Oct 30 21:42:42.257675 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:42.257655    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ebcd709c-90f1-4a0f-828e-676364b15b93-tmp\") pod \"downloads-6dd789c8b7-4s89b\" (UID: \"ebcd709c-90f1-4a0f-828e-676364b15b93\") " pod="openshift-console/downloads-6dd789c8b7-4s89b"
Oct 30 21:42:42.257737 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:42.257723    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfrgc\" (UniqueName: \"kubernetes.io/projected/ebcd709c-90f1-4a0f-828e-676364b15b93-kube-api-access-zfrgc\") pod \"downloads-6dd789c8b7-4s89b\" (UID: \"ebcd709c-90f1-4a0f-828e-676364b15b93\") " pod="openshift-console/downloads-6dd789c8b7-4s89b"
Oct 30 21:42:42.315615 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:42.315586    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5d4d6cb75d-hrc4d"]
Oct 30 21:42:42.329138 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:42.329070    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-4s89b"]
Oct 30 21:42:42.358598 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:42.358568    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/59e77938-67b5-4d51-be2b-62cc0999b427-bound-sa-token\") pod \"image-registry-5d4d6cb75d-hrc4d\" (UID: \"59e77938-67b5-4d51-be2b-62cc0999b427\") " pod="openshift-image-registry/image-registry-5d4d6cb75d-hrc4d"
Oct 30 21:42:42.358598 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:42.358601    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfxqc\" (UniqueName: \"kubernetes.io/projected/59e77938-67b5-4d51-be2b-62cc0999b427-kube-api-access-cfxqc\") pod \"image-registry-5d4d6cb75d-hrc4d\" (UID: \"59e77938-67b5-4d51-be2b-62cc0999b427\") " pod="openshift-image-registry/image-registry-5d4d6cb75d-hrc4d"
Oct 30 21:42:42.358747 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:42.358622    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/59e77938-67b5-4d51-be2b-62cc0999b427-installation-pull-secrets\") pod \"image-registry-5d4d6cb75d-hrc4d\" (UID: \"59e77938-67b5-4d51-be2b-62cc0999b427\") " pod="openshift-image-registry/image-registry-5d4d6cb75d-hrc4d"
Oct 30 21:42:42.358747 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:42.358647    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/59e77938-67b5-4d51-be2b-62cc0999b427-registry-certificates\") pod \"image-registry-5d4d6cb75d-hrc4d\" (UID: \"59e77938-67b5-4d51-be2b-62cc0999b427\") " pod="openshift-image-registry/image-registry-5d4d6cb75d-hrc4d"
Oct 30 21:42:42.358747 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:42.358736    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfrgc\" (UniqueName: \"kubernetes.io/projected/ebcd709c-90f1-4a0f-828e-676364b15b93-kube-api-access-zfrgc\") pod \"downloads-6dd789c8b7-4s89b\" (UID: \"ebcd709c-90f1-4a0f-828e-676364b15b93\") " pod="openshift-console/downloads-6dd789c8b7-4s89b"
Oct 30 21:42:42.358837 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:42.358808    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/59e77938-67b5-4d51-be2b-62cc0999b427-registry-tls\") pod \"image-registry-5d4d6cb75d-hrc4d\" (UID: \"59e77938-67b5-4d51-be2b-62cc0999b427\") " pod="openshift-image-registry/image-registry-5d4d6cb75d-hrc4d"
Oct 30 21:42:42.358870 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:42.358857    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/6339c2e5-a2c9-4e70-88e6-97285a01a8ff-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-8pk6d\" (UID: \"6339c2e5-a2c9-4e70-88e6-97285a01a8ff\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-8pk6d"
Oct 30 21:42:42.358899 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:42.358884    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/59e77938-67b5-4d51-be2b-62cc0999b427-trusted-ca\") pod \"image-registry-5d4d6cb75d-hrc4d\" (UID: \"59e77938-67b5-4d51-be2b-62cc0999b427\") " pod="openshift-image-registry/image-registry-5d4d6cb75d-hrc4d"
Oct 30 21:42:42.358949 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:42.358932    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/59e77938-67b5-4d51-be2b-62cc0999b427-image-registry-private-configuration\") pod \"image-registry-5d4d6cb75d-hrc4d\" (UID: \"59e77938-67b5-4d51-be2b-62cc0999b427\") " pod="openshift-image-registry/image-registry-5d4d6cb75d-hrc4d"
Oct 30 21:42:42.359052 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:42.359033    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/59e77938-67b5-4d51-be2b-62cc0999b427-ca-trust-extracted\") pod \"image-registry-5d4d6cb75d-hrc4d\" (UID: \"59e77938-67b5-4d51-be2b-62cc0999b427\") " pod="openshift-image-registry/image-registry-5d4d6cb75d-hrc4d"
Oct 30 21:42:42.359099 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:42.359085    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ebcd709c-90f1-4a0f-828e-676364b15b93-tmp\") pod \"downloads-6dd789c8b7-4s89b\" (UID: \"ebcd709c-90f1-4a0f-828e-676364b15b93\") " pod="openshift-console/downloads-6dd789c8b7-4s89b"
Oct 30 21:42:42.359436 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:42.359412    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ebcd709c-90f1-4a0f-828e-676364b15b93-tmp\") pod \"downloads-6dd789c8b7-4s89b\" (UID: \"ebcd709c-90f1-4a0f-828e-676364b15b93\") " pod="openshift-console/downloads-6dd789c8b7-4s89b"
Oct 30 21:42:42.361235 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:42.361214    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/6339c2e5-a2c9-4e70-88e6-97285a01a8ff-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-8pk6d\" (UID: \"6339c2e5-a2c9-4e70-88e6-97285a01a8ff\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-8pk6d"
Oct 30 21:42:42.415866 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:42.415838    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfrgc\" (UniqueName: \"kubernetes.io/projected/ebcd709c-90f1-4a0f-828e-676364b15b93-kube-api-access-zfrgc\") pod \"downloads-6dd789c8b7-4s89b\" (UID: \"ebcd709c-90f1-4a0f-828e-676364b15b93\") " pod="openshift-console/downloads-6dd789c8b7-4s89b"
Oct 30 21:42:42.422997 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:42.422979    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-4s89b"
Oct 30 21:42:42.448215 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:42.448192    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-8pk6d"
Oct 30 21:42:42.460280 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:42.460253    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/59e77938-67b5-4d51-be2b-62cc0999b427-trusted-ca\") pod \"image-registry-5d4d6cb75d-hrc4d\" (UID: \"59e77938-67b5-4d51-be2b-62cc0999b427\") " pod="openshift-image-registry/image-registry-5d4d6cb75d-hrc4d"
Oct 30 21:42:42.460381 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:42.460311    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/59e77938-67b5-4d51-be2b-62cc0999b427-image-registry-private-configuration\") pod \"image-registry-5d4d6cb75d-hrc4d\" (UID: \"59e77938-67b5-4d51-be2b-62cc0999b427\") " pod="openshift-image-registry/image-registry-5d4d6cb75d-hrc4d"
Oct 30 21:42:42.460381 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:42.460364    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/59e77938-67b5-4d51-be2b-62cc0999b427-ca-trust-extracted\") pod \"image-registry-5d4d6cb75d-hrc4d\" (UID: \"59e77938-67b5-4d51-be2b-62cc0999b427\") " pod="openshift-image-registry/image-registry-5d4d6cb75d-hrc4d"
Oct 30 21:42:42.460478 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:42.460406    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/59e77938-67b5-4d51-be2b-62cc0999b427-bound-sa-token\") pod \"image-registry-5d4d6cb75d-hrc4d\" (UID: \"59e77938-67b5-4d51-be2b-62cc0999b427\") " pod="openshift-image-registry/image-registry-5d4d6cb75d-hrc4d"
Oct 30 21:42:42.460478 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:42.460436    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfxqc\" (UniqueName: \"kubernetes.io/projected/59e77938-67b5-4d51-be2b-62cc0999b427-kube-api-access-cfxqc\") pod \"image-registry-5d4d6cb75d-hrc4d\" (UID: \"59e77938-67b5-4d51-be2b-62cc0999b427\") " pod="openshift-image-registry/image-registry-5d4d6cb75d-hrc4d"
Oct 30 21:42:42.460478 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:42.460468    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/59e77938-67b5-4d51-be2b-62cc0999b427-installation-pull-secrets\") pod \"image-registry-5d4d6cb75d-hrc4d\" (UID: \"59e77938-67b5-4d51-be2b-62cc0999b427\") " pod="openshift-image-registry/image-registry-5d4d6cb75d-hrc4d"
Oct 30 21:42:42.460672 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:42.460504    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/59e77938-67b5-4d51-be2b-62cc0999b427-registry-certificates\") pod \"image-registry-5d4d6cb75d-hrc4d\" (UID: \"59e77938-67b5-4d51-be2b-62cc0999b427\") " pod="openshift-image-registry/image-registry-5d4d6cb75d-hrc4d"
Oct 30 21:42:42.460672 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:42.460549    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/59e77938-67b5-4d51-be2b-62cc0999b427-registry-tls\") pod \"image-registry-5d4d6cb75d-hrc4d\" (UID: \"59e77938-67b5-4d51-be2b-62cc0999b427\") " pod="openshift-image-registry/image-registry-5d4d6cb75d-hrc4d"
Oct 30 21:42:42.461975 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:42.461946    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/59e77938-67b5-4d51-be2b-62cc0999b427-registry-certificates\") pod \"image-registry-5d4d6cb75d-hrc4d\" (UID: \"59e77938-67b5-4d51-be2b-62cc0999b427\") " pod="openshift-image-registry/image-registry-5d4d6cb75d-hrc4d"
Oct 30 21:42:42.462730 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:42.462706    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/59e77938-67b5-4d51-be2b-62cc0999b427-trusted-ca\") pod \"image-registry-5d4d6cb75d-hrc4d\" (UID: \"59e77938-67b5-4d51-be2b-62cc0999b427\") " pod="openshift-image-registry/image-registry-5d4d6cb75d-hrc4d"
Oct 30 21:42:42.463137 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:42.463113    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/59e77938-67b5-4d51-be2b-62cc0999b427-ca-trust-extracted\") pod \"image-registry-5d4d6cb75d-hrc4d\" (UID: \"59e77938-67b5-4d51-be2b-62cc0999b427\") " pod="openshift-image-registry/image-registry-5d4d6cb75d-hrc4d"
Oct 30 21:42:42.463879 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:42.463856    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/59e77938-67b5-4d51-be2b-62cc0999b427-registry-tls\") pod \"image-registry-5d4d6cb75d-hrc4d\" (UID: \"59e77938-67b5-4d51-be2b-62cc0999b427\") " pod="openshift-image-registry/image-registry-5d4d6cb75d-hrc4d"
Oct 30 21:42:42.464870 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:42.464845    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/59e77938-67b5-4d51-be2b-62cc0999b427-image-registry-private-configuration\") pod \"image-registry-5d4d6cb75d-hrc4d\" (UID: \"59e77938-67b5-4d51-be2b-62cc0999b427\") " pod="openshift-image-registry/image-registry-5d4d6cb75d-hrc4d"
Oct 30 21:42:42.465206 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:42.465187    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/59e77938-67b5-4d51-be2b-62cc0999b427-installation-pull-secrets\") pod \"image-registry-5d4d6cb75d-hrc4d\" (UID: \"59e77938-67b5-4d51-be2b-62cc0999b427\") " pod="openshift-image-registry/image-registry-5d4d6cb75d-hrc4d"
Oct 30 21:42:42.518270 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:42.518222    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/59e77938-67b5-4d51-be2b-62cc0999b427-bound-sa-token\") pod \"image-registry-5d4d6cb75d-hrc4d\" (UID: \"59e77938-67b5-4d51-be2b-62cc0999b427\") " pod="openshift-image-registry/image-registry-5d4d6cb75d-hrc4d"
Oct 30 21:42:42.520993 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:42.520958    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfxqc\" (UniqueName: \"kubernetes.io/projected/59e77938-67b5-4d51-be2b-62cc0999b427-kube-api-access-cfxqc\") pod \"image-registry-5d4d6cb75d-hrc4d\" (UID: \"59e77938-67b5-4d51-be2b-62cc0999b427\") " pod="openshift-image-registry/image-registry-5d4d6cb75d-hrc4d"
Oct 30 21:42:42.539211 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:42.539189    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d4d6cb75d-hrc4d"
Oct 30 21:42:42.818791 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:42.818760    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-4s89b"]
Oct 30 21:42:42.821366 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:42.821341    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebcd709c_90f1_4a0f_828e_676364b15b93.slice/crio-b457ba1b9f2cc7b5f0fd79595c702eb3998cb05571db36e94e018449c879c1df WatchSource:0}: Error finding container b457ba1b9f2cc7b5f0fd79595c702eb3998cb05571db36e94e018449c879c1df: Status 404 returned error can't find the container with id b457ba1b9f2cc7b5f0fd79595c702eb3998cb05571db36e94e018449c879c1df
Oct 30 21:42:42.831331 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:42.831236    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-8pk6d"]
Oct 30 21:42:42.833200 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:42.833177    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6339c2e5_a2c9_4e70_88e6_97285a01a8ff.slice/crio-76c84f2cb63810e8ef962c25cf2accf7fbe9cf1d85085657492fb5dcdadfb0a7 WatchSource:0}: Error finding container 76c84f2cb63810e8ef962c25cf2accf7fbe9cf1d85085657492fb5dcdadfb0a7: Status 404 returned error can't find the container with id 76c84f2cb63810e8ef962c25cf2accf7fbe9cf1d85085657492fb5dcdadfb0a7
Oct 30 21:42:42.904970 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:42.904951    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5d4d6cb75d-hrc4d"]
Oct 30 21:42:42.906262 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:42.906233    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59e77938_67b5_4d51_be2b_62cc0999b427.slice/crio-4b5b91e5c6a96ccdb81ebd1bc483ba5dcf48f6f1f3b67525fcf781c4e07e6654 WatchSource:0}: Error finding container 4b5b91e5c6a96ccdb81ebd1bc483ba5dcf48f6f1f3b67525fcf781c4e07e6654: Status 404 returned error can't find the container with id 4b5b91e5c6a96ccdb81ebd1bc483ba5dcf48f6f1f3b67525fcf781c4e07e6654
Oct 30 21:42:43.469794 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:43.469757    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/74ff988a-c05e-4a3e-adff-5945b79cf248-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mstdj\" (UID: \"74ff988a-c05e-4a3e-adff-5945b79cf248\") " pod="openshift-insights/insights-runtime-extractor-mstdj"
Oct 30 21:42:43.472257 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:43.472227    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/74ff988a-c05e-4a3e-adff-5945b79cf248-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mstdj\" (UID: \"74ff988a-c05e-4a3e-adff-5945b79cf248\") " pod="openshift-insights/insights-runtime-extractor-mstdj"
Oct 30 21:42:43.526103 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:43.526075    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-mstdj"
Oct 30 21:42:43.704244 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:43.704210    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5d4d6cb75d-hrc4d" event={"ID":"59e77938-67b5-4d51-be2b-62cc0999b427","Type":"ContainerStarted","Data":"877133c85993d85621a8d8f0244c64b97615edbf8f78a409ec42d79c6d650fe0"}
Oct 30 21:42:43.704415 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:43.704273    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5d4d6cb75d-hrc4d"
Oct 30 21:42:43.704415 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:43.704293    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5d4d6cb75d-hrc4d" event={"ID":"59e77938-67b5-4d51-be2b-62cc0999b427","Type":"ContainerStarted","Data":"4b5b91e5c6a96ccdb81ebd1bc483ba5dcf48f6f1f3b67525fcf781c4e07e6654"}
Oct 30 21:42:43.705377 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:43.705355    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-4s89b" event={"ID":"ebcd709c-90f1-4a0f-828e-676364b15b93","Type":"ContainerStarted","Data":"b457ba1b9f2cc7b5f0fd79595c702eb3998cb05571db36e94e018449c879c1df"}
Oct 30 21:42:43.706414 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:43.706393    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-8pk6d" event={"ID":"6339c2e5-a2c9-4e70-88e6-97285a01a8ff","Type":"ContainerStarted","Data":"76c84f2cb63810e8ef962c25cf2accf7fbe9cf1d85085657492fb5dcdadfb0a7"}
Oct 30 21:42:43.816848 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:43.816793    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5d4d6cb75d-hrc4d" podStartSLOduration=1.816778115 podStartE2EDuration="1.816778115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:42:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:43.816268753 +0000 UTC m=+17.429038531" watchObservedRunningTime="2026-10-30 21:42:43.816778115 +0000 UTC m=+17.429547873"
Oct 30 21:42:43.925630 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:43.925598    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-mstdj"]
Oct 30 21:42:43.930153 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:43.930113    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74ff988a_c05e_4a3e_adff_5945b79cf248.slice/crio-a42291eb52497b5c2e8f2baf83b4cec732f42bcc3cc44be39d310058fdae7da0 WatchSource:0}: Error finding container a42291eb52497b5c2e8f2baf83b4cec732f42bcc3cc44be39d310058fdae7da0: Status 404 returned error can't find the container with id a42291eb52497b5c2e8f2baf83b4cec732f42bcc3cc44be39d310058fdae7da0
Oct 30 21:42:44.713678 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:44.713495    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-8pk6d" event={"ID":"6339c2e5-a2c9-4e70-88e6-97285a01a8ff","Type":"ContainerStarted","Data":"f287046f21f0371c9aa10ae365417bee4c77ff251caeeb8e377fab34ac6f2d7b"}
Oct 30 21:42:44.713678 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:44.713585    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-8pk6d"
Oct 30 21:42:44.716394 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:44.716361    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mstdj" event={"ID":"74ff988a-c05e-4a3e-adff-5945b79cf248","Type":"ContainerStarted","Data":"18ee75932ef0452df47bca0ea492e2cd634bbfb9af5cd6294d0207b6763dbe3f"}
Oct 30 21:42:44.716553 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:44.716417    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mstdj" event={"ID":"74ff988a-c05e-4a3e-adff-5945b79cf248","Type":"ContainerStarted","Data":"a42291eb52497b5c2e8f2baf83b4cec732f42bcc3cc44be39d310058fdae7da0"}
Oct 30 21:42:44.719700 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:44.719674    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-8pk6d"
Oct 30 21:42:44.822391 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:44.822337    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-8pk6d" podStartSLOduration=1.27189156 podStartE2EDuration="2.822319097s" totalImagesPullingTime="1.550427537s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:42:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:44.821626245 +0000 UTC m=+18.434396003" watchObservedRunningTime="2026-10-30 21:42:44.822319097 +0000 UTC m=+18.435088856"
Oct 30 21:42:45.725640 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:45.725565    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mstdj" event={"ID":"74ff988a-c05e-4a3e-adff-5945b79cf248","Type":"ContainerStarted","Data":"f545003232b08d6b43b7a9ddd09eff90d39f4d1b883fa3db32f093b1abc0cd94"}
Oct 30 21:42:47.743811 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:47.743714    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mstdj" event={"ID":"74ff988a-c05e-4a3e-adff-5945b79cf248","Type":"ContainerStarted","Data":"d3088007ae38fa081ddd5666542fbd7dac7ed1fd951f6f70cd080da75b632551"}
Oct 30 21:42:48.639871 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:48.639828    5061 patch_prober.go:29] interesting pod/image-registry-b6cdbd45d-bdftd container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Oct 30 21:42:48.640042 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:48.639895    5061 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-b6cdbd45d-bdftd" podUID="cad67bc5-c8d2-497c-ba1b-cdbae5822cfa" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 21:42:49.693735 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:49.693698    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-6pttx"
Oct 30 21:42:50.939179 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:50.939151    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-22ldv"
Oct 30 21:42:52.130117 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:52.130088    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-d44c8cbb7-lmw97"
Oct 30 21:42:52.839508 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:52.839471    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-pqv7p"]
Oct 30 21:42:52.844450 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:52.844424    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-pqv7p"
Oct 30 21:42:52.856628 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:52.856595    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0090a21f-28fc-4b66-aef7-e4a23e118236-node-exporter-tls\") pod \"node-exporter-pqv7p\" (UID: \"0090a21f-28fc-4b66-aef7-e4a23e118236\") " pod="openshift-monitoring/node-exporter-pqv7p"
Oct 30 21:42:52.856758 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:52.856646    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0090a21f-28fc-4b66-aef7-e4a23e118236-sys\") pod \"node-exporter-pqv7p\" (UID: \"0090a21f-28fc-4b66-aef7-e4a23e118236\") " pod="openshift-monitoring/node-exporter-pqv7p"
Oct 30 21:42:52.856758 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:52.856672    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0090a21f-28fc-4b66-aef7-e4a23e118236-node-exporter-wtmp\") pod \"node-exporter-pqv7p\" (UID: \"0090a21f-28fc-4b66-aef7-e4a23e118236\") " pod="openshift-monitoring/node-exporter-pqv7p"
Oct 30 21:42:52.856758 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:52.856723    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0090a21f-28fc-4b66-aef7-e4a23e118236-metrics-client-ca\") pod \"node-exporter-pqv7p\" (UID: \"0090a21f-28fc-4b66-aef7-e4a23e118236\") " pod="openshift-monitoring/node-exporter-pqv7p"
Oct 30 21:42:52.856882 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:52.856807    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0090a21f-28fc-4b66-aef7-e4a23e118236-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pqv7p\" (UID: \"0090a21f-28fc-4b66-aef7-e4a23e118236\") " pod="openshift-monitoring/node-exporter-pqv7p"
Oct 30 21:42:52.856882 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:52.856874    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0090a21f-28fc-4b66-aef7-e4a23e118236-root\") pod \"node-exporter-pqv7p\" (UID: \"0090a21f-28fc-4b66-aef7-e4a23e118236\") " pod="openshift-monitoring/node-exporter-pqv7p"
Oct 30 21:42:52.856952 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:52.856908    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0090a21f-28fc-4b66-aef7-e4a23e118236-node-exporter-accelerators-collector-config\") pod \"node-exporter-pqv7p\" (UID: \"0090a21f-28fc-4b66-aef7-e4a23e118236\") " pod="openshift-monitoring/node-exporter-pqv7p"
Oct 30 21:42:52.856952 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:52.856938    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kc6p\" (UniqueName: \"kubernetes.io/projected/0090a21f-28fc-4b66-aef7-e4a23e118236-kube-api-access-2kc6p\") pod \"node-exporter-pqv7p\" (UID: \"0090a21f-28fc-4b66-aef7-e4a23e118236\") " pod="openshift-monitoring/node-exporter-pqv7p"
Oct 30 21:42:52.857029 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:52.856987    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0090a21f-28fc-4b66-aef7-e4a23e118236-node-exporter-textfile\") pod \"node-exporter-pqv7p\" (UID: \"0090a21f-28fc-4b66-aef7-e4a23e118236\") " pod="openshift-monitoring/node-exporter-pqv7p"
Oct 30 21:42:52.894543 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:52.894494    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 30 21:42:52.941940 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:52.941878    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 30 21:42:52.942233 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:52.942195    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 30 21:42:52.942233 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:52.942218    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 30 21:42:52.942441 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:52.942286    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-hv9kv\""
Oct 30 21:42:52.957733 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:52.957705    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0090a21f-28fc-4b66-aef7-e4a23e118236-node-exporter-tls\") pod \"node-exporter-pqv7p\" (UID: \"0090a21f-28fc-4b66-aef7-e4a23e118236\") " pod="openshift-monitoring/node-exporter-pqv7p"
Oct 30 21:42:52.957850 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:52.957750    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0090a21f-28fc-4b66-aef7-e4a23e118236-sys\") pod \"node-exporter-pqv7p\" (UID: \"0090a21f-28fc-4b66-aef7-e4a23e118236\") " pod="openshift-monitoring/node-exporter-pqv7p"
Oct 30 21:42:52.957850 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:52.957803    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0090a21f-28fc-4b66-aef7-e4a23e118236-sys\") pod \"node-exporter-pqv7p\" (UID: \"0090a21f-28fc-4b66-aef7-e4a23e118236\") " pod="openshift-monitoring/node-exporter-pqv7p"
Oct 30 21:42:52.957850 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:52.957822    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0090a21f-28fc-4b66-aef7-e4a23e118236-node-exporter-wtmp\") pod \"node-exporter-pqv7p\" (UID: \"0090a21f-28fc-4b66-aef7-e4a23e118236\") " pod="openshift-monitoring/node-exporter-pqv7p"
Oct 30 21:42:52.957850 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:52.957845    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0090a21f-28fc-4b66-aef7-e4a23e118236-metrics-client-ca\") pod \"node-exporter-pqv7p\" (UID: \"0090a21f-28fc-4b66-aef7-e4a23e118236\") " pod="openshift-monitoring/node-exporter-pqv7p"
Oct 30 21:42:52.958016 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:52.957875    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0090a21f-28fc-4b66-aef7-e4a23e118236-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pqv7p\" (UID: \"0090a21f-28fc-4b66-aef7-e4a23e118236\") " pod="openshift-monitoring/node-exporter-pqv7p"
Oct 30 21:42:52.958016 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:52.957938    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0090a21f-28fc-4b66-aef7-e4a23e118236-root\") pod \"node-exporter-pqv7p\" (UID: \"0090a21f-28fc-4b66-aef7-e4a23e118236\") " pod="openshift-monitoring/node-exporter-pqv7p"
Oct 30 21:42:52.958016 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:52.957966    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0090a21f-28fc-4b66-aef7-e4a23e118236-node-exporter-accelerators-collector-config\") pod \"node-exporter-pqv7p\" (UID: \"0090a21f-28fc-4b66-aef7-e4a23e118236\") " pod="openshift-monitoring/node-exporter-pqv7p"
Oct 30 21:42:52.958016 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:52.957983    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0090a21f-28fc-4b66-aef7-e4a23e118236-node-exporter-wtmp\") pod \"node-exporter-pqv7p\" (UID: \"0090a21f-28fc-4b66-aef7-e4a23e118236\") " pod="openshift-monitoring/node-exporter-pqv7p"
Oct 30 21:42:52.958016 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:52.957993    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kc6p\" (UniqueName: \"kubernetes.io/projected/0090a21f-28fc-4b66-aef7-e4a23e118236-kube-api-access-2kc6p\") pod \"node-exporter-pqv7p\" (UID: \"0090a21f-28fc-4b66-aef7-e4a23e118236\") " pod="openshift-monitoring/node-exporter-pqv7p"
Oct 30 21:42:52.958211 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:52.958046    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0090a21f-28fc-4b66-aef7-e4a23e118236-node-exporter-textfile\") pod \"node-exporter-pqv7p\" (UID: \"0090a21f-28fc-4b66-aef7-e4a23e118236\") " pod="openshift-monitoring/node-exporter-pqv7p"
Oct 30 21:42:52.958211 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:52.958061    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0090a21f-28fc-4b66-aef7-e4a23e118236-root\") pod \"node-exporter-pqv7p\" (UID: \"0090a21f-28fc-4b66-aef7-e4a23e118236\") " pod="openshift-monitoring/node-exporter-pqv7p"
Oct 30 21:42:52.958454 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:52.958420    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0090a21f-28fc-4b66-aef7-e4a23e118236-node-exporter-textfile\") pod \"node-exporter-pqv7p\" (UID: \"0090a21f-28fc-4b66-aef7-e4a23e118236\") " pod="openshift-monitoring/node-exporter-pqv7p"
Oct 30 21:42:52.958641 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:52.958619    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0090a21f-28fc-4b66-aef7-e4a23e118236-metrics-client-ca\") pod \"node-exporter-pqv7p\" (UID: \"0090a21f-28fc-4b66-aef7-e4a23e118236\") " pod="openshift-monitoring/node-exporter-pqv7p"
Oct 30 21:42:52.958730 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:52.958621    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0090a21f-28fc-4b66-aef7-e4a23e118236-node-exporter-accelerators-collector-config\") pod \"node-exporter-pqv7p\" (UID: \"0090a21f-28fc-4b66-aef7-e4a23e118236\") " pod="openshift-monitoring/node-exporter-pqv7p"
Oct 30 21:42:52.960359 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:52.960338    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0090a21f-28fc-4b66-aef7-e4a23e118236-node-exporter-tls\") pod \"node-exporter-pqv7p\" (UID: \"0090a21f-28fc-4b66-aef7-e4a23e118236\") " pod="openshift-monitoring/node-exporter-pqv7p"
Oct 30 21:42:52.960719 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:52.960701    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0090a21f-28fc-4b66-aef7-e4a23e118236-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pqv7p\" (UID: \"0090a21f-28fc-4b66-aef7-e4a23e118236\") " pod="openshift-monitoring/node-exporter-pqv7p"
Oct 30 21:42:53.023036 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:53.023004    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kc6p\" (UniqueName: \"kubernetes.io/projected/0090a21f-28fc-4b66-aef7-e4a23e118236-kube-api-access-2kc6p\") pod \"node-exporter-pqv7p\" (UID: \"0090a21f-28fc-4b66-aef7-e4a23e118236\") " pod="openshift-monitoring/node-exporter-pqv7p"
Oct 30 21:42:53.162692 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:53.162617    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-pqv7p"
Oct 30 21:42:53.177798 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:42:53.177767    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0090a21f_28fc_4b66_aef7_e4a23e118236.slice/crio-40a19420d3ec465fafb61852a7654a6b1d2d9417bde1fe35a4d876277fe1995b WatchSource:0}: Error finding container 40a19420d3ec465fafb61852a7654a6b1d2d9417bde1fe35a4d876277fe1995b: Status 404 returned error can't find the container with id 40a19420d3ec465fafb61852a7654a6b1d2d9417bde1fe35a4d876277fe1995b
Oct 30 21:42:53.773183 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:53.773144    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pqv7p" event={"ID":"0090a21f-28fc-4b66-aef7-e4a23e118236","Type":"ContainerStarted","Data":"40a19420d3ec465fafb61852a7654a6b1d2d9417bde1fe35a4d876277fe1995b"}
Oct 30 21:42:53.801851 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:53.801823    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-b6cdbd45d-bdftd"
Oct 30 21:42:54.673507 ip-10-0-130-60 kubenswrapper[5061]: E1030 21:42:54.673467    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 21:42:55.788123 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:55.788089    5061 generic.go:356] "Generic (PLEG): container finished" podID="0090a21f-28fc-4b66-aef7-e4a23e118236" containerID="69b99baa7c481023e8dabf997b40c467232197c5d1eb0bef597055b523e37c53" exitCode=0
Oct 30 21:42:55.788562 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:55.788179    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pqv7p" event={"ID":"0090a21f-28fc-4b66-aef7-e4a23e118236","Type":"ContainerDied","Data":"69b99baa7c481023e8dabf997b40c467232197c5d1eb0bef597055b523e37c53"}
Oct 30 21:42:56.796218 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:56.796180    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pqv7p" event={"ID":"0090a21f-28fc-4b66-aef7-e4a23e118236","Type":"ContainerStarted","Data":"565579291f4db5587daa1288d7399ce6a26493824eccf31908f749600377e83a"}
Oct 30 21:42:56.796664 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:56.796246    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pqv7p" event={"ID":"0090a21f-28fc-4b66-aef7-e4a23e118236","Type":"ContainerStarted","Data":"a3006d5be8798ed0448196a21ed333962a13951d88085919392a5ef69662dbff"}
Oct 30 21:42:56.956628 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:56.956572    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-pqv7p" podStartSLOduration=3.242661792 podStartE2EDuration="4.956555044s" totalImagesPullingTime="1.636994152s" totalInitContainerRuntime="76.8991ms" isStatefulPod=true podCreationTimestamp="2026-10-30 21:42:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:56.955961398 +0000 UTC m=+30.568731156" watchObservedRunningTime="2026-10-30 21:42:56.956555044 +0000 UTC m=+30.569324803"
Oct 30 21:42:58.799912 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:42:58.799876    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-979r9"
Oct 30 21:43:00.820687 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:43:00.820644    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-4s89b" event={"ID":"ebcd709c-90f1-4a0f-828e-676364b15b93","Type":"ContainerStarted","Data":"9252f520dfba0a70f26cd19731799a7cdd38449e5a53f564044f9faf47f5f8a8"}
Oct 30 21:43:00.821178 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:43:00.820943    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-4s89b"
Oct 30 21:43:00.835493 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:43:00.835465    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-4s89b"
Oct 30 21:43:01.026461 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:43:01.026397    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-4s89b" podStartSLOduration=1.728895829 podStartE2EDuration="19.02637781s" totalImagesPullingTime="17.297481981s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:42:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:43:00.977712611 +0000 UTC m=+34.590482369" watchObservedRunningTime="2026-10-30 21:43:01.02637781 +0000 UTC m=+34.639147569"
Oct 30 21:43:04.721187 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:43:04.721158    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5d4d6cb75d-hrc4d"
Oct 30 21:43:04.933417 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:43:04.933379    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-b6cdbd45d-bdftd"]
Oct 30 21:43:07.149905 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:43:07.149828    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-d44c8cbb7-lmw97" podUID="613e7c8a-e2c7-424a-93e0-7fa86d18a8a9" containerName="registry" containerID="cri-o://0aaa71c56c29f494a17f99a4237e93c1f698b737a3e83a9a3a0aca110714021d" gracePeriod=30
Oct 30 21:43:07.444198 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:43:07.444170    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-d44c8cbb7-lmw97"
Oct 30 21:43:07.485812 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:43:07.485770    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/613e7c8a-e2c7-424a-93e0-7fa86d18a8a9-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/613e7c8a-e2c7-424a-93e0-7fa86d18a8a9-installation-pull-secrets\") pod \"613e7c8a-e2c7-424a-93e0-7fa86d18a8a9\" (UID: \"613e7c8a-e2c7-424a-93e0-7fa86d18a8a9\") "
Oct 30 21:43:07.485979 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:43:07.485838    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/613e7c8a-e2c7-424a-93e0-7fa86d18a8a9-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/613e7c8a-e2c7-424a-93e0-7fa86d18a8a9-ca-trust-extracted\") pod \"613e7c8a-e2c7-424a-93e0-7fa86d18a8a9\" (UID: \"613e7c8a-e2c7-424a-93e0-7fa86d18a8a9\") "
Oct 30 21:43:07.485979 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:43:07.485878    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/613e7c8a-e2c7-424a-93e0-7fa86d18a8a9-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/613e7c8a-e2c7-424a-93e0-7fa86d18a8a9-registry-certificates\") pod \"613e7c8a-e2c7-424a-93e0-7fa86d18a8a9\" (UID: \"613e7c8a-e2c7-424a-93e0-7fa86d18a8a9\") "
Oct 30 21:43:07.485979 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:43:07.485909    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/613e7c8a-e2c7-424a-93e0-7fa86d18a8a9-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/613e7c8a-e2c7-424a-93e0-7fa86d18a8a9-image-registry-private-configuration\") pod \"613e7c8a-e2c7-424a-93e0-7fa86d18a8a9\" (UID: \"613e7c8a-e2c7-424a-93e0-7fa86d18a8a9\") "
Oct 30 21:43:07.485979 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:43:07.485945    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/613e7c8a-e2c7-424a-93e0-7fa86d18a8a9-kube-api-access-hw226\" (UniqueName: \"kubernetes.io/projected/613e7c8a-e2c7-424a-93e0-7fa86d18a8a9-kube-api-access-hw226\") pod \"613e7c8a-e2c7-424a-93e0-7fa86d18a8a9\" (UID: \"613e7c8a-e2c7-424a-93e0-7fa86d18a8a9\") "
Oct 30 21:43:07.486197 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:43:07.485991    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/613e7c8a-e2c7-424a-93e0-7fa86d18a8a9-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/613e7c8a-e2c7-424a-93e0-7fa86d18a8a9-trusted-ca\") pod \"613e7c8a-e2c7-424a-93e0-7fa86d18a8a9\" (UID: \"613e7c8a-e2c7-424a-93e0-7fa86d18a8a9\") "
Oct 30 21:43:07.486197 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:43:07.486021    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/613e7c8a-e2c7-424a-93e0-7fa86d18a8a9-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/613e7c8a-e2c7-424a-93e0-7fa86d18a8a9-bound-sa-token\") pod \"613e7c8a-e2c7-424a-93e0-7fa86d18a8a9\" (UID: \"613e7c8a-e2c7-424a-93e0-7fa86d18a8a9\") "
Oct 30 21:43:07.486197 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:43:07.486051    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/613e7c8a-e2c7-424a-93e0-7fa86d18a8a9-registry-tls\" (UniqueName: \"kubernetes.io/projected/613e7c8a-e2c7-424a-93e0-7fa86d18a8a9-registry-tls\") pod \"613e7c8a-e2c7-424a-93e0-7fa86d18a8a9\" (UID: \"613e7c8a-e2c7-424a-93e0-7fa86d18a8a9\") "
Oct 30 21:43:07.486365 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:43:07.486342    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/613e7c8a-e2c7-424a-93e0-7fa86d18a8a9-registry-certificates" pod "613e7c8a-e2c7-424a-93e0-7fa86d18a8a9" (UID: "613e7c8a-e2c7-424a-93e0-7fa86d18a8a9"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:43:07.486871 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:43:07.486828    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/613e7c8a-e2c7-424a-93e0-7fa86d18a8a9-trusted-ca" pod "613e7c8a-e2c7-424a-93e0-7fa86d18a8a9" (UID: "613e7c8a-e2c7-424a-93e0-7fa86d18a8a9"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:43:07.490286 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:43:07.490259    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/613e7c8a-e2c7-424a-93e0-7fa86d18a8a9-installation-pull-secrets" pod "613e7c8a-e2c7-424a-93e0-7fa86d18a8a9" (UID: "613e7c8a-e2c7-424a-93e0-7fa86d18a8a9"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:43:07.490433 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:43:07.490294    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/613e7c8a-e2c7-424a-93e0-7fa86d18a8a9-bound-sa-token" pod "613e7c8a-e2c7-424a-93e0-7fa86d18a8a9" (UID: "613e7c8a-e2c7-424a-93e0-7fa86d18a8a9"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:43:07.490503 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:43:07.490431    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/613e7c8a-e2c7-424a-93e0-7fa86d18a8a9-image-registry-private-configuration" pod "613e7c8a-e2c7-424a-93e0-7fa86d18a8a9" (UID: "613e7c8a-e2c7-424a-93e0-7fa86d18a8a9"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:43:07.494681 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:43:07.494651    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/613e7c8a-e2c7-424a-93e0-7fa86d18a8a9-registry-tls" pod "613e7c8a-e2c7-424a-93e0-7fa86d18a8a9" (UID: "613e7c8a-e2c7-424a-93e0-7fa86d18a8a9"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:43:07.494774 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:43:07.494666    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/613e7c8a-e2c7-424a-93e0-7fa86d18a8a9-kube-api-access-hw226" pod "613e7c8a-e2c7-424a-93e0-7fa86d18a8a9" (UID: "613e7c8a-e2c7-424a-93e0-7fa86d18a8a9"). InnerVolumeSpecName "kube-api-access-hw226". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:43:07.494831 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:43:07.494802    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/613e7c8a-e2c7-424a-93e0-7fa86d18a8a9-ca-trust-extracted" pod "613e7c8a-e2c7-424a-93e0-7fa86d18a8a9" (UID: "613e7c8a-e2c7-424a-93e0-7fa86d18a8a9"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:43:07.586868 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:43:07.586836    5061 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/613e7c8a-e2c7-424a-93e0-7fa86d18a8a9-installation-pull-secrets\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:43:07.586868 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:43:07.586869    5061 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/613e7c8a-e2c7-424a-93e0-7fa86d18a8a9-ca-trust-extracted\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:43:07.587027 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:43:07.586880    5061 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/613e7c8a-e2c7-424a-93e0-7fa86d18a8a9-registry-certificates\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:43:07.587027 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:43:07.586889    5061 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/613e7c8a-e2c7-424a-93e0-7fa86d18a8a9-image-registry-private-configuration\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:43:07.587027 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:43:07.586899    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hw226\" (UniqueName: \"kubernetes.io/projected/613e7c8a-e2c7-424a-93e0-7fa86d18a8a9-kube-api-access-hw226\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:43:07.587027 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:43:07.586912    5061 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/613e7c8a-e2c7-424a-93e0-7fa86d18a8a9-trusted-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:43:07.587027 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:43:07.586926    5061 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/613e7c8a-e2c7-424a-93e0-7fa86d18a8a9-bound-sa-token\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:43:07.587027 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:43:07.586940    5061 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/613e7c8a-e2c7-424a-93e0-7fa86d18a8a9-registry-tls\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:43:07.855235 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:43:07.855195    5061 generic.go:356] "Generic (PLEG): container finished" podID="613e7c8a-e2c7-424a-93e0-7fa86d18a8a9" containerID="0aaa71c56c29f494a17f99a4237e93c1f698b737a3e83a9a3a0aca110714021d" exitCode=0
Oct 30 21:43:07.855415 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:43:07.855252    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-d44c8cbb7-lmw97"
Oct 30 21:43:07.855415 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:43:07.855286    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-d44c8cbb7-lmw97" event={"ID":"613e7c8a-e2c7-424a-93e0-7fa86d18a8a9","Type":"ContainerDied","Data":"0aaa71c56c29f494a17f99a4237e93c1f698b737a3e83a9a3a0aca110714021d"}
Oct 30 21:43:07.855415 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:43:07.855338    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-d44c8cbb7-lmw97" event={"ID":"613e7c8a-e2c7-424a-93e0-7fa86d18a8a9","Type":"ContainerDied","Data":"cf44e7310f7a985eddb19ab993be5a1e859fbae93f90c65ec347d84f93c73eae"}
Oct 30 21:43:07.855415 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:43:07.855365    5061 scope.go:122] "RemoveContainer" containerID="0aaa71c56c29f494a17f99a4237e93c1f698b737a3e83a9a3a0aca110714021d"
Oct 30 21:43:07.866126 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:43:07.866100    5061 scope.go:122] "RemoveContainer" containerID="0aaa71c56c29f494a17f99a4237e93c1f698b737a3e83a9a3a0aca110714021d"
Oct 30 21:43:07.866434 ip-10-0-130-60 kubenswrapper[5061]: E1030 21:43:07.866408    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0aaa71c56c29f494a17f99a4237e93c1f698b737a3e83a9a3a0aca110714021d\": container with ID starting with 0aaa71c56c29f494a17f99a4237e93c1f698b737a3e83a9a3a0aca110714021d not found: ID does not exist" containerID="0aaa71c56c29f494a17f99a4237e93c1f698b737a3e83a9a3a0aca110714021d"
Oct 30 21:43:07.866551 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:43:07.866444    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0aaa71c56c29f494a17f99a4237e93c1f698b737a3e83a9a3a0aca110714021d"} err="failed to get container status \"0aaa71c56c29f494a17f99a4237e93c1f698b737a3e83a9a3a0aca110714021d\": rpc error: code = NotFound desc = could not find container \"0aaa71c56c29f494a17f99a4237e93c1f698b737a3e83a9a3a0aca110714021d\": container with ID starting with 0aaa71c56c29f494a17f99a4237e93c1f698b737a3e83a9a3a0aca110714021d not found: ID does not exist"
Oct 30 21:43:08.018476 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:43:08.018442    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-d44c8cbb7-lmw97"]
Oct 30 21:43:08.024931 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:43:08.024904    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-d44c8cbb7-lmw97"]
Oct 30 21:43:08.527606 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:43:08.527567    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="613e7c8a-e2c7-424a-93e0-7fa86d18a8a9" path="/var/lib/kubelet/pods/613e7c8a-e2c7-424a-93e0-7fa86d18a8a9/volumes"
Oct 30 21:43:11.023021 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:43:11.022987    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sztmx_238ff7d8-7da8-4cd8-9d45-deedf615211e/konnectivity-agent/0.log" limit=1024
Oct 30 21:43:11.453663 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:43:11.453610    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sztmx_238ff7d8-7da8-4cd8-9d45-deedf615211e/konnectivity-agent/0.log" limit=1024
Oct 30 21:43:24.665475 ip-10-0-130-60 kubenswrapper[5061]: E1030 21:43:24.665433    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 21:43:29.956921 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:43:29.956871    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-b6cdbd45d-bdftd" podUID="cad67bc5-c8d2-497c-ba1b-cdbae5822cfa" containerName="registry" containerID="cri-o://425394fcf72ef30f2fc6caca30666a86ae8e21732166003e89071dc2e5089383" gracePeriod=30
Oct 30 21:43:30.227778 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:43:30.227712    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-b6cdbd45d-bdftd"
Oct 30 21:43:30.383072 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:43:30.383045    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cad67bc5-c8d2-497c-ba1b-cdbae5822cfa-kube-api-access-m9dpv\" (UniqueName: \"kubernetes.io/projected/cad67bc5-c8d2-497c-ba1b-cdbae5822cfa-kube-api-access-m9dpv\") pod \"cad67bc5-c8d2-497c-ba1b-cdbae5822cfa\" (UID: \"cad67bc5-c8d2-497c-ba1b-cdbae5822cfa\") "
Oct 30 21:43:30.383072 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:43:30.383077    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cad67bc5-c8d2-497c-ba1b-cdbae5822cfa-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cad67bc5-c8d2-497c-ba1b-cdbae5822cfa-image-registry-private-configuration\") pod \"cad67bc5-c8d2-497c-ba1b-cdbae5822cfa\" (UID: \"cad67bc5-c8d2-497c-ba1b-cdbae5822cfa\") "
Oct 30 21:43:30.383348 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:43:30.383095    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cad67bc5-c8d2-497c-ba1b-cdbae5822cfa-registry-tls\" (UniqueName: \"kubernetes.io/projected/cad67bc5-c8d2-497c-ba1b-cdbae5822cfa-registry-tls\") pod \"cad67bc5-c8d2-497c-ba1b-cdbae5822cfa\" (UID: \"cad67bc5-c8d2-497c-ba1b-cdbae5822cfa\") "
Oct 30 21:43:30.383348 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:43:30.383117    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cad67bc5-c8d2-497c-ba1b-cdbae5822cfa-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cad67bc5-c8d2-497c-ba1b-cdbae5822cfa-bound-sa-token\") pod \"cad67bc5-c8d2-497c-ba1b-cdbae5822cfa\" (UID: \"cad67bc5-c8d2-497c-ba1b-cdbae5822cfa\") "
Oct 30 21:43:30.383348 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:43:30.383258    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cad67bc5-c8d2-497c-ba1b-cdbae5822cfa-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cad67bc5-c8d2-497c-ba1b-cdbae5822cfa-trusted-ca\") pod \"cad67bc5-c8d2-497c-ba1b-cdbae5822cfa\" (UID: \"cad67bc5-c8d2-497c-ba1b-cdbae5822cfa\") "
Oct 30 21:43:30.383348 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:43:30.383314    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cad67bc5-c8d2-497c-ba1b-cdbae5822cfa-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cad67bc5-c8d2-497c-ba1b-cdbae5822cfa-installation-pull-secrets\") pod \"cad67bc5-c8d2-497c-ba1b-cdbae5822cfa\" (UID: \"cad67bc5-c8d2-497c-ba1b-cdbae5822cfa\") "
Oct 30 21:43:30.383593 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:43:30.383352    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cad67bc5-c8d2-497c-ba1b-cdbae5822cfa-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cad67bc5-c8d2-497c-ba1b-cdbae5822cfa-registry-certificates\") pod \"cad67bc5-c8d2-497c-ba1b-cdbae5822cfa\" (UID: \"cad67bc5-c8d2-497c-ba1b-cdbae5822cfa\") "
Oct 30 21:43:30.383593 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:43:30.383399    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cad67bc5-c8d2-497c-ba1b-cdbae5822cfa-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cad67bc5-c8d2-497c-ba1b-cdbae5822cfa-ca-trust-extracted\") pod \"cad67bc5-c8d2-497c-ba1b-cdbae5822cfa\" (UID: \"cad67bc5-c8d2-497c-ba1b-cdbae5822cfa\") "
Oct 30 21:43:30.383964 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:43:30.383907    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cad67bc5-c8d2-497c-ba1b-cdbae5822cfa-trusted-ca" pod "cad67bc5-c8d2-497c-ba1b-cdbae5822cfa" (UID: "cad67bc5-c8d2-497c-ba1b-cdbae5822cfa"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:43:30.384056 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:43:30.383970    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cad67bc5-c8d2-497c-ba1b-cdbae5822cfa-registry-certificates" pod "cad67bc5-c8d2-497c-ba1b-cdbae5822cfa" (UID: "cad67bc5-c8d2-497c-ba1b-cdbae5822cfa"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:43:30.385459 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:43:30.385432    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cad67bc5-c8d2-497c-ba1b-cdbae5822cfa-registry-tls" pod "cad67bc5-c8d2-497c-ba1b-cdbae5822cfa" (UID: "cad67bc5-c8d2-497c-ba1b-cdbae5822cfa"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:43:30.385724 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:43:30.385685    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cad67bc5-c8d2-497c-ba1b-cdbae5822cfa-bound-sa-token" pod "cad67bc5-c8d2-497c-ba1b-cdbae5822cfa" (UID: "cad67bc5-c8d2-497c-ba1b-cdbae5822cfa"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:43:30.385824 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:43:30.385801    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cad67bc5-c8d2-497c-ba1b-cdbae5822cfa-kube-api-access-m9dpv" pod "cad67bc5-c8d2-497c-ba1b-cdbae5822cfa" (UID: "cad67bc5-c8d2-497c-ba1b-cdbae5822cfa"). InnerVolumeSpecName "kube-api-access-m9dpv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:43:30.385890 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:43:30.385832    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cad67bc5-c8d2-497c-ba1b-cdbae5822cfa-image-registry-private-configuration" pod "cad67bc5-c8d2-497c-ba1b-cdbae5822cfa" (UID: "cad67bc5-c8d2-497c-ba1b-cdbae5822cfa"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:43:30.386015 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:43:30.385998    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cad67bc5-c8d2-497c-ba1b-cdbae5822cfa-installation-pull-secrets" pod "cad67bc5-c8d2-497c-ba1b-cdbae5822cfa" (UID: "cad67bc5-c8d2-497c-ba1b-cdbae5822cfa"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:43:30.390692 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:43:30.390578    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cad67bc5-c8d2-497c-ba1b-cdbae5822cfa-ca-trust-extracted" pod "cad67bc5-c8d2-497c-ba1b-cdbae5822cfa" (UID: "cad67bc5-c8d2-497c-ba1b-cdbae5822cfa"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:43:30.484467 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:43:30.484367    5061 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cad67bc5-c8d2-497c-ba1b-cdbae5822cfa-installation-pull-secrets\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:43:30.484467 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:43:30.484406    5061 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cad67bc5-c8d2-497c-ba1b-cdbae5822cfa-registry-certificates\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:43:30.484467 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:43:30.484417    5061 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cad67bc5-c8d2-497c-ba1b-cdbae5822cfa-ca-trust-extracted\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:43:30.484467 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:43:30.484427    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9dpv\" (UniqueName: \"kubernetes.io/projected/cad67bc5-c8d2-497c-ba1b-cdbae5822cfa-kube-api-access-m9dpv\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:43:30.484467 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:43:30.484436    5061 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cad67bc5-c8d2-497c-ba1b-cdbae5822cfa-image-registry-private-configuration\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:43:30.484467 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:43:30.484445    5061 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cad67bc5-c8d2-497c-ba1b-cdbae5822cfa-registry-tls\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:43:30.484467 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:43:30.484455    5061 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cad67bc5-c8d2-497c-ba1b-cdbae5822cfa-bound-sa-token\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:43:30.484467 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:43:30.484464    5061 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cad67bc5-c8d2-497c-ba1b-cdbae5822cfa-trusted-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:43:30.950415 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:43:30.950373    5061 generic.go:356] "Generic (PLEG): container finished" podID="cad67bc5-c8d2-497c-ba1b-cdbae5822cfa" containerID="425394fcf72ef30f2fc6caca30666a86ae8e21732166003e89071dc2e5089383" exitCode=0
Oct 30 21:43:30.950663 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:43:30.950429    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-b6cdbd45d-bdftd"
Oct 30 21:43:30.950663 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:43:30.950458    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-b6cdbd45d-bdftd" event={"ID":"cad67bc5-c8d2-497c-ba1b-cdbae5822cfa","Type":"ContainerDied","Data":"425394fcf72ef30f2fc6caca30666a86ae8e21732166003e89071dc2e5089383"}
Oct 30 21:43:30.950663 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:43:30.950507    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-b6cdbd45d-bdftd" event={"ID":"cad67bc5-c8d2-497c-ba1b-cdbae5822cfa","Type":"ContainerDied","Data":"5fc41a9f51634495988a3328fa35ed11319cfa44aa6fce95ebb4811cdb77cddd"}
Oct 30 21:43:30.950663 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:43:30.950545    5061 scope.go:122] "RemoveContainer" containerID="425394fcf72ef30f2fc6caca30666a86ae8e21732166003e89071dc2e5089383"
Oct 30 21:43:30.958892 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:43:30.958601    5061 scope.go:122] "RemoveContainer" containerID="425394fcf72ef30f2fc6caca30666a86ae8e21732166003e89071dc2e5089383"
Oct 30 21:43:30.958892 ip-10-0-130-60 kubenswrapper[5061]: E1030 21:43:30.958857    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"425394fcf72ef30f2fc6caca30666a86ae8e21732166003e89071dc2e5089383\": container with ID starting with 425394fcf72ef30f2fc6caca30666a86ae8e21732166003e89071dc2e5089383 not found: ID does not exist" containerID="425394fcf72ef30f2fc6caca30666a86ae8e21732166003e89071dc2e5089383"
Oct 30 21:43:30.958892 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:43:30.958883    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"425394fcf72ef30f2fc6caca30666a86ae8e21732166003e89071dc2e5089383"} err="failed to get container status \"425394fcf72ef30f2fc6caca30666a86ae8e21732166003e89071dc2e5089383\": rpc error: code = NotFound desc = could not find container \"425394fcf72ef30f2fc6caca30666a86ae8e21732166003e89071dc2e5089383\": container with ID starting with 425394fcf72ef30f2fc6caca30666a86ae8e21732166003e89071dc2e5089383 not found: ID does not exist"
Oct 30 21:43:31.156548 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:43:31.156498    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-b6cdbd45d-bdftd"]
Oct 30 21:43:31.204110 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:43:31.204041    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-b6cdbd45d-bdftd"]
Oct 30 21:43:31.956091 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:43:31.956062    5061 generic.go:356] "Generic (PLEG): container finished" podID="c69312da-8be3-4931-8a81-a8ff3f86d5e6" containerID="ca9e120be9fab54658452c365699191ee58804aef4e46c11271dd00227394e4b" exitCode=0
Oct 30 21:43:31.956261 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:43:31.956145    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-l9b9d" event={"ID":"c69312da-8be3-4931-8a81-a8ff3f86d5e6","Type":"ContainerDied","Data":"ca9e120be9fab54658452c365699191ee58804aef4e46c11271dd00227394e4b"}
Oct 30 21:43:31.956473 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:43:31.956459    5061 scope.go:122] "RemoveContainer" containerID="ca9e120be9fab54658452c365699191ee58804aef4e46c11271dd00227394e4b"
Oct 30 21:43:32.526459 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:43:32.526430    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cad67bc5-c8d2-497c-ba1b-cdbae5822cfa" path="/var/lib/kubelet/pods/cad67bc5-c8d2-497c-ba1b-cdbae5822cfa/volumes"
Oct 30 21:43:32.904046 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:43:32.904016    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-7t4kc-b65p7"]
Oct 30 21:43:32.904217 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:43:32.904198    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-7t4kc-b65p7" podUID="101c697b-d7b8-4cfe-9d4c-818b5c04108f" containerName="insights-gathering" containerID="cri-o://b8fc53580d7eec0aa43f21a2e71f286fdde0bb9b422c8c9afdbfee8ea539aadc" gracePeriod=30
Oct 30 21:43:32.961890 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:43:32.961856    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-l9b9d" event={"ID":"c69312da-8be3-4931-8a81-a8ff3f86d5e6","Type":"ContainerStarted","Data":"8a50c021c878d3474277db642a13f21d673937760dadd316742a93b6968367e3"}
Oct 30 21:43:33.201175 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:43:33.201145    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-7t4kc-b65p7"
Oct 30 21:43:33.309657 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:43:33.309623    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/101c697b-d7b8-4cfe-9d4c-818b5c04108f-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/101c697b-d7b8-4cfe-9d4c-818b5c04108f-service-ca-bundle\") pod \"101c697b-d7b8-4cfe-9d4c-818b5c04108f\" (UID: \"101c697b-d7b8-4cfe-9d4c-818b5c04108f\") "
Oct 30 21:43:33.309828 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:43:33.309694    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/101c697b-d7b8-4cfe-9d4c-818b5c04108f-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/101c697b-d7b8-4cfe-9d4c-818b5c04108f-trusted-ca-bundle\") pod \"101c697b-d7b8-4cfe-9d4c-818b5c04108f\" (UID: \"101c697b-d7b8-4cfe-9d4c-818b5c04108f\") "
Oct 30 21:43:33.309828 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:43:33.309782    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/101c697b-d7b8-4cfe-9d4c-818b5c04108f-kube-api-access-xm5l5\" (UniqueName: \"kubernetes.io/projected/101c697b-d7b8-4cfe-9d4c-818b5c04108f-kube-api-access-xm5l5\") pod \"101c697b-d7b8-4cfe-9d4c-818b5c04108f\" (UID: \"101c697b-d7b8-4cfe-9d4c-818b5c04108f\") "
Oct 30 21:43:33.309917 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:43:33.309828    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/101c697b-d7b8-4cfe-9d4c-818b5c04108f-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/101c697b-d7b8-4cfe-9d4c-818b5c04108f-archives-path\") pod \"101c697b-d7b8-4cfe-9d4c-818b5c04108f\" (UID: \"101c697b-d7b8-4cfe-9d4c-818b5c04108f\") "
Oct 30 21:43:33.310059 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:43:33.309996    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/101c697b-d7b8-4cfe-9d4c-818b5c04108f-service-ca-bundle" pod "101c697b-d7b8-4cfe-9d4c-818b5c04108f" (UID: "101c697b-d7b8-4cfe-9d4c-818b5c04108f"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:43:33.310120 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:43:33.310094    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/101c697b-d7b8-4cfe-9d4c-818b5c04108f-trusted-ca-bundle" pod "101c697b-d7b8-4cfe-9d4c-818b5c04108f" (UID: "101c697b-d7b8-4cfe-9d4c-818b5c04108f"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:43:33.310157 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:43:33.310141    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/101c697b-d7b8-4cfe-9d4c-818b5c04108f-archives-path" pod "101c697b-d7b8-4cfe-9d4c-818b5c04108f" (UID: "101c697b-d7b8-4cfe-9d4c-818b5c04108f"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:43:33.312064 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:43:33.312036    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/101c697b-d7b8-4cfe-9d4c-818b5c04108f-kube-api-access-xm5l5" pod "101c697b-d7b8-4cfe-9d4c-818b5c04108f" (UID: "101c697b-d7b8-4cfe-9d4c-818b5c04108f"). InnerVolumeSpecName "kube-api-access-xm5l5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:43:33.410751 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:43:33.410675    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xm5l5\" (UniqueName: \"kubernetes.io/projected/101c697b-d7b8-4cfe-9d4c-818b5c04108f-kube-api-access-xm5l5\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:43:33.410751 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:43:33.410704    5061 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/101c697b-d7b8-4cfe-9d4c-818b5c04108f-archives-path\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:43:33.410751 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:43:33.410714    5061 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/101c697b-d7b8-4cfe-9d4c-818b5c04108f-service-ca-bundle\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:43:33.410751 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:43:33.410723    5061 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/101c697b-d7b8-4cfe-9d4c-818b5c04108f-trusted-ca-bundle\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:43:33.971126 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:43:33.971093    5061 generic.go:356] "Generic (PLEG): container finished" podID="101c697b-d7b8-4cfe-9d4c-818b5c04108f" containerID="b8fc53580d7eec0aa43f21a2e71f286fdde0bb9b422c8c9afdbfee8ea539aadc" exitCode=2
Oct 30 21:43:33.971653 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:43:33.971161    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-7t4kc-b65p7"
Oct 30 21:43:33.971653 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:43:33.971163    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-7t4kc-b65p7" event={"ID":"101c697b-d7b8-4cfe-9d4c-818b5c04108f","Type":"ContainerDied","Data":"b8fc53580d7eec0aa43f21a2e71f286fdde0bb9b422c8c9afdbfee8ea539aadc"}
Oct 30 21:43:33.971653 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:43:33.971297    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-7t4kc-b65p7" event={"ID":"101c697b-d7b8-4cfe-9d4c-818b5c04108f","Type":"ContainerDied","Data":"c25682648a4da3674e83bed18c8a7c4f12a24442876464de11bece9d61180c06"}
Oct 30 21:43:33.971653 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:43:33.971317    5061 scope.go:122] "RemoveContainer" containerID="b8fc53580d7eec0aa43f21a2e71f286fdde0bb9b422c8c9afdbfee8ea539aadc"
Oct 30 21:43:33.979475 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:43:33.979457    5061 scope.go:122] "RemoveContainer" containerID="b8fc53580d7eec0aa43f21a2e71f286fdde0bb9b422c8c9afdbfee8ea539aadc"
Oct 30 21:43:33.979725 ip-10-0-130-60 kubenswrapper[5061]: E1030 21:43:33.979704    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8fc53580d7eec0aa43f21a2e71f286fdde0bb9b422c8c9afdbfee8ea539aadc\": container with ID starting with b8fc53580d7eec0aa43f21a2e71f286fdde0bb9b422c8c9afdbfee8ea539aadc not found: ID does not exist" containerID="b8fc53580d7eec0aa43f21a2e71f286fdde0bb9b422c8c9afdbfee8ea539aadc"
Oct 30 21:43:33.979779 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:43:33.979734    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8fc53580d7eec0aa43f21a2e71f286fdde0bb9b422c8c9afdbfee8ea539aadc"} err="failed to get container status \"b8fc53580d7eec0aa43f21a2e71f286fdde0bb9b422c8c9afdbfee8ea539aadc\": rpc error: code = NotFound desc = could not find container \"b8fc53580d7eec0aa43f21a2e71f286fdde0bb9b422c8c9afdbfee8ea539aadc\": container with ID starting with b8fc53580d7eec0aa43f21a2e71f286fdde0bb9b422c8c9afdbfee8ea539aadc not found: ID does not exist"
Oct 30 21:43:34.133124 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:43:34.133092    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-7t4kc-b65p7"]
Oct 30 21:43:34.180284 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:43:34.180253    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-7t4kc-b65p7"]
Oct 30 21:43:34.525921 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:43:34.525892    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="101c697b-d7b8-4cfe-9d4c-818b5c04108f" path="/var/lib/kubelet/pods/101c697b-d7b8-4cfe-9d4c-818b5c04108f/volumes"
Oct 30 21:43:54.664632 ip-10-0-130-60 kubenswrapper[5061]: E1030 21:43:54.664596    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 21:44:24.668609 ip-10-0-130-60 kubenswrapper[5061]: E1030 21:44:24.668570    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 21:44:54.665705 ip-10-0-130-60 kubenswrapper[5061]: E1030 21:44:54.665627    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 21:45:24.665869 ip-10-0-130-60 kubenswrapper[5061]: E1030 21:45:24.665835    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 21:45:54.665007 ip-10-0-130-60 kubenswrapper[5061]: E1030 21:45:54.664966    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 21:45:59.949193 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:45:59.949160    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-58dhn"]
Oct 30 21:45:59.949604 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:45:59.949460    5061 cpu_manager.go:414] "containerMap: removing container" podUID="cad67bc5-c8d2-497c-ba1b-cdbae5822cfa" containerName="registry"
Oct 30 21:45:59.949604 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:45:59.949470    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cad67bc5-c8d2-497c-ba1b-cdbae5822cfa" containerName="registry"
Oct 30 21:45:59.949604 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:45:59.949491    5061 cpu_manager.go:414] "containerMap: removing container" podUID="101c697b-d7b8-4cfe-9d4c-818b5c04108f" containerName="insights-gathering"
Oct 30 21:45:59.949604 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:45:59.949496    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="101c697b-d7b8-4cfe-9d4c-818b5c04108f" containerName="insights-gathering"
Oct 30 21:45:59.949604 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:45:59.949507    5061 cpu_manager.go:414] "containerMap: removing container" podUID="613e7c8a-e2c7-424a-93e0-7fa86d18a8a9" containerName="registry"
Oct 30 21:45:59.949604 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:45:59.949512    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="613e7c8a-e2c7-424a-93e0-7fa86d18a8a9" containerName="registry"
Oct 30 21:45:59.949604 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:45:59.949595    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="cad67bc5-c8d2-497c-ba1b-cdbae5822cfa" containerName="registry"
Oct 30 21:45:59.949604 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:45:59.949606    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="101c697b-d7b8-4cfe-9d4c-818b5c04108f" containerName="insights-gathering"
Oct 30 21:45:59.949852 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:45:59.949613    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="613e7c8a-e2c7-424a-93e0-7fa86d18a8a9" containerName="registry"
Oct 30 21:45:59.951735 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:45:59.951713    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-58dhn"
Oct 30 21:46:00.001260 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:00.001233    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 30 21:46:00.028131 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:00.028107    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-d6th4\""
Oct 30 21:46:00.056340 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:00.056306    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-58dhn"]
Oct 30 21:46:00.086372 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:00.086349    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99439df0-9c24-4a03-9dff-e95af6f3f3bd-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-58dhn\" (UID: \"99439df0-9c24-4a03-9dff-e95af6f3f3bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-58dhn"
Oct 30 21:46:00.086492 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:00.086390    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99439df0-9c24-4a03-9dff-e95af6f3f3bd-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-58dhn\" (UID: \"99439df0-9c24-4a03-9dff-e95af6f3f3bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-58dhn"
Oct 30 21:46:00.086492 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:00.086411    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99439df0-9c24-4a03-9dff-e95af6f3f3bd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-58dhn\" (UID: \"99439df0-9c24-4a03-9dff-e95af6f3f3bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-58dhn"
Oct 30 21:46:00.086631 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:00.086492    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99439df0-9c24-4a03-9dff-e95af6f3f3bd-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-58dhn\" (UID: \"99439df0-9c24-4a03-9dff-e95af6f3f3bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-58dhn"
Oct 30 21:46:00.086631 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:00.086595    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99439df0-9c24-4a03-9dff-e95af6f3f3bd-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-58dhn\" (UID: \"99439df0-9c24-4a03-9dff-e95af6f3f3bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-58dhn"
Oct 30 21:46:00.086734 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:00.086635    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99439df0-9c24-4a03-9dff-e95af6f3f3bd-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-58dhn\" (UID: \"99439df0-9c24-4a03-9dff-e95af6f3f3bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-58dhn"
Oct 30 21:46:00.086734 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:00.086663    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99439df0-9c24-4a03-9dff-e95af6f3f3bd-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-58dhn\" (UID: \"99439df0-9c24-4a03-9dff-e95af6f3f3bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-58dhn"
Oct 30 21:46:00.086734 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:00.086707    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsgst\" (UniqueName: \"kubernetes.io/projected/99439df0-9c24-4a03-9dff-e95af6f3f3bd-kube-api-access-qsgst\") pod \"maas-default-gateway-openshift-default-694db55c87-58dhn\" (UID: \"99439df0-9c24-4a03-9dff-e95af6f3f3bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-58dhn"
Oct 30 21:46:00.086880 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:00.086739    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99439df0-9c24-4a03-9dff-e95af6f3f3bd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-58dhn\" (UID: \"99439df0-9c24-4a03-9dff-e95af6f3f3bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-58dhn"
Oct 30 21:46:00.187188 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:00.187162    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99439df0-9c24-4a03-9dff-e95af6f3f3bd-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-58dhn\" (UID: \"99439df0-9c24-4a03-9dff-e95af6f3f3bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-58dhn"
Oct 30 21:46:00.187270 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:00.187203    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99439df0-9c24-4a03-9dff-e95af6f3f3bd-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-58dhn\" (UID: \"99439df0-9c24-4a03-9dff-e95af6f3f3bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-58dhn"
Oct 30 21:46:00.187270 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:00.187220    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99439df0-9c24-4a03-9dff-e95af6f3f3bd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-58dhn\" (UID: \"99439df0-9c24-4a03-9dff-e95af6f3f3bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-58dhn"
Oct 30 21:46:00.187270 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:00.187238    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99439df0-9c24-4a03-9dff-e95af6f3f3bd-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-58dhn\" (UID: \"99439df0-9c24-4a03-9dff-e95af6f3f3bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-58dhn"
Oct 30 21:46:00.187270 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:00.187261    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99439df0-9c24-4a03-9dff-e95af6f3f3bd-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-58dhn\" (UID: \"99439df0-9c24-4a03-9dff-e95af6f3f3bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-58dhn"
Oct 30 21:46:00.187477 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:00.187284    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99439df0-9c24-4a03-9dff-e95af6f3f3bd-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-58dhn\" (UID: \"99439df0-9c24-4a03-9dff-e95af6f3f3bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-58dhn"
Oct 30 21:46:00.187477 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:00.187317    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99439df0-9c24-4a03-9dff-e95af6f3f3bd-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-58dhn\" (UID: \"99439df0-9c24-4a03-9dff-e95af6f3f3bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-58dhn"
Oct 30 21:46:00.187617 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:00.187566    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsgst\" (UniqueName: \"kubernetes.io/projected/99439df0-9c24-4a03-9dff-e95af6f3f3bd-kube-api-access-qsgst\") pod \"maas-default-gateway-openshift-default-694db55c87-58dhn\" (UID: \"99439df0-9c24-4a03-9dff-e95af6f3f3bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-58dhn"
Oct 30 21:46:00.187685 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:00.187623    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99439df0-9c24-4a03-9dff-e95af6f3f3bd-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-58dhn\" (UID: \"99439df0-9c24-4a03-9dff-e95af6f3f3bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-58dhn"
Oct 30 21:46:00.187685 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:00.187626    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99439df0-9c24-4a03-9dff-e95af6f3f3bd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-58dhn\" (UID: \"99439df0-9c24-4a03-9dff-e95af6f3f3bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-58dhn"
Oct 30 21:46:00.187685 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:00.187670    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99439df0-9c24-4a03-9dff-e95af6f3f3bd-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-58dhn\" (UID: \"99439df0-9c24-4a03-9dff-e95af6f3f3bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-58dhn"
Oct 30 21:46:00.187833 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:00.187774    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99439df0-9c24-4a03-9dff-e95af6f3f3bd-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-58dhn\" (UID: \"99439df0-9c24-4a03-9dff-e95af6f3f3bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-58dhn"
Oct 30 21:46:00.187833 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:00.187791    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99439df0-9c24-4a03-9dff-e95af6f3f3bd-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-58dhn\" (UID: \"99439df0-9c24-4a03-9dff-e95af6f3f3bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-58dhn"
Oct 30 21:46:00.188040 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:00.188022    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99439df0-9c24-4a03-9dff-e95af6f3f3bd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-58dhn\" (UID: \"99439df0-9c24-4a03-9dff-e95af6f3f3bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-58dhn"
Oct 30 21:46:00.189676 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:00.189655    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99439df0-9c24-4a03-9dff-e95af6f3f3bd-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-58dhn\" (UID: \"99439df0-9c24-4a03-9dff-e95af6f3f3bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-58dhn"
Oct 30 21:46:00.189869 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:00.189853    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99439df0-9c24-4a03-9dff-e95af6f3f3bd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-58dhn\" (UID: \"99439df0-9c24-4a03-9dff-e95af6f3f3bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-58dhn"
Oct 30 21:46:00.240880 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:00.240825    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99439df0-9c24-4a03-9dff-e95af6f3f3bd-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-58dhn\" (UID: \"99439df0-9c24-4a03-9dff-e95af6f3f3bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-58dhn"
Oct 30 21:46:00.240880 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:00.240872    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsgst\" (UniqueName: \"kubernetes.io/projected/99439df0-9c24-4a03-9dff-e95af6f3f3bd-kube-api-access-qsgst\") pod \"maas-default-gateway-openshift-default-694db55c87-58dhn\" (UID: \"99439df0-9c24-4a03-9dff-e95af6f3f3bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-58dhn"
Oct 30 21:46:00.261514 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:00.261491    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-4lt8x"]
Oct 30 21:46:00.262539 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:00.262515    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-58dhn"
Oct 30 21:46:00.264287 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:00.264272    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-4lt8x"
Oct 30 21:46:00.314879 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:00.314363    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-bz85c\""
Oct 30 21:46:00.327349 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:00.327278    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-4lt8x"]
Oct 30 21:46:00.389749 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:00.389720    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24dtn\" (UniqueName: \"kubernetes.io/projected/36808b98-fda2-40eb-86a0-923a27e268ac-kube-api-access-24dtn\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-4lt8x\" (UID: \"36808b98-fda2-40eb-86a0-923a27e268ac\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-4lt8x"
Oct 30 21:46:00.389882 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:00.389761    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36808b98-fda2-40eb-86a0-923a27e268ac-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-4lt8x\" (UID: \"36808b98-fda2-40eb-86a0-923a27e268ac\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-4lt8x"
Oct 30 21:46:00.389882 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:00.389783    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36808b98-fda2-40eb-86a0-923a27e268ac-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-4lt8x\" (UID: \"36808b98-fda2-40eb-86a0-923a27e268ac\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-4lt8x"
Oct 30 21:46:00.389882 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:00.389872    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36808b98-fda2-40eb-86a0-923a27e268ac-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-4lt8x\" (UID: \"36808b98-fda2-40eb-86a0-923a27e268ac\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-4lt8x"
Oct 30 21:46:00.389986 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:00.389891    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36808b98-fda2-40eb-86a0-923a27e268ac-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-4lt8x\" (UID: \"36808b98-fda2-40eb-86a0-923a27e268ac\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-4lt8x"
Oct 30 21:46:00.389986 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:00.389910    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36808b98-fda2-40eb-86a0-923a27e268ac-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-4lt8x\" (UID: \"36808b98-fda2-40eb-86a0-923a27e268ac\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-4lt8x"
Oct 30 21:46:00.390054 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:00.389983    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36808b98-fda2-40eb-86a0-923a27e268ac-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-4lt8x\" (UID: \"36808b98-fda2-40eb-86a0-923a27e268ac\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-4lt8x"
Oct 30 21:46:00.390054 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:00.390019    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36808b98-fda2-40eb-86a0-923a27e268ac-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-4lt8x\" (UID: \"36808b98-fda2-40eb-86a0-923a27e268ac\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-4lt8x"
Oct 30 21:46:00.390118 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:00.390051    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36808b98-fda2-40eb-86a0-923a27e268ac-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-4lt8x\" (UID: \"36808b98-fda2-40eb-86a0-923a27e268ac\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-4lt8x"
Oct 30 21:46:00.490615 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:00.490583    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36808b98-fda2-40eb-86a0-923a27e268ac-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-4lt8x\" (UID: \"36808b98-fda2-40eb-86a0-923a27e268ac\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-4lt8x"
Oct 30 21:46:00.490615 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:00.490617    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36808b98-fda2-40eb-86a0-923a27e268ac-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-4lt8x\" (UID: \"36808b98-fda2-40eb-86a0-923a27e268ac\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-4lt8x"
Oct 30 21:46:00.490810 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:00.490632    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36808b98-fda2-40eb-86a0-923a27e268ac-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-4lt8x\" (UID: \"36808b98-fda2-40eb-86a0-923a27e268ac\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-4lt8x"
Oct 30 21:46:00.490810 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:00.490683    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24dtn\" (UniqueName: \"kubernetes.io/projected/36808b98-fda2-40eb-86a0-923a27e268ac-kube-api-access-24dtn\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-4lt8x\" (UID: \"36808b98-fda2-40eb-86a0-923a27e268ac\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-4lt8x"
Oct 30 21:46:00.490810 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:00.490706    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36808b98-fda2-40eb-86a0-923a27e268ac-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-4lt8x\" (UID: \"36808b98-fda2-40eb-86a0-923a27e268ac\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-4lt8x"
Oct 30 21:46:00.490810 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:00.490743    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36808b98-fda2-40eb-86a0-923a27e268ac-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-4lt8x\" (UID: \"36808b98-fda2-40eb-86a0-923a27e268ac\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-4lt8x"
Oct 30 21:46:00.490810 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:00.490767    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36808b98-fda2-40eb-86a0-923a27e268ac-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-4lt8x\" (UID: \"36808b98-fda2-40eb-86a0-923a27e268ac\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-4lt8x"
Oct 30 21:46:00.490810 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:00.490794    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36808b98-fda2-40eb-86a0-923a27e268ac-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-4lt8x\" (UID: \"36808b98-fda2-40eb-86a0-923a27e268ac\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-4lt8x"
Oct 30 21:46:00.491158 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:00.490824    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36808b98-fda2-40eb-86a0-923a27e268ac-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-4lt8x\" (UID: \"36808b98-fda2-40eb-86a0-923a27e268ac\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-4lt8x"
Oct 30 21:46:00.491158 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:00.491004    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36808b98-fda2-40eb-86a0-923a27e268ac-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-4lt8x\" (UID: \"36808b98-fda2-40eb-86a0-923a27e268ac\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-4lt8x"
Oct 30 21:46:00.491560 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:00.491514    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36808b98-fda2-40eb-86a0-923a27e268ac-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-4lt8x\" (UID: \"36808b98-fda2-40eb-86a0-923a27e268ac\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-4lt8x"
Oct 30 21:46:00.491687 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:00.491563    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36808b98-fda2-40eb-86a0-923a27e268ac-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-4lt8x\" (UID: \"36808b98-fda2-40eb-86a0-923a27e268ac\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-4lt8x"
Oct 30 21:46:00.491687 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:00.491600    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36808b98-fda2-40eb-86a0-923a27e268ac-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-4lt8x\" (UID: \"36808b98-fda2-40eb-86a0-923a27e268ac\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-4lt8x"
Oct 30 21:46:00.491687 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:00.491651    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36808b98-fda2-40eb-86a0-923a27e268ac-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-4lt8x\" (UID: \"36808b98-fda2-40eb-86a0-923a27e268ac\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-4lt8x"
Oct 30 21:46:00.493126 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:00.493108    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36808b98-fda2-40eb-86a0-923a27e268ac-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-4lt8x\" (UID: \"36808b98-fda2-40eb-86a0-923a27e268ac\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-4lt8x"
Oct 30 21:46:00.493225 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:00.493208    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36808b98-fda2-40eb-86a0-923a27e268ac-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-4lt8x\" (UID: \"36808b98-fda2-40eb-86a0-923a27e268ac\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-4lt8x"
Oct 30 21:46:00.545413 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:00.545386    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36808b98-fda2-40eb-86a0-923a27e268ac-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-4lt8x\" (UID: \"36808b98-fda2-40eb-86a0-923a27e268ac\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-4lt8x"
Oct 30 21:46:00.545614 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:00.545598    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24dtn\" (UniqueName: \"kubernetes.io/projected/36808b98-fda2-40eb-86a0-923a27e268ac-kube-api-access-24dtn\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-4lt8x\" (UID: \"36808b98-fda2-40eb-86a0-923a27e268ac\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-4lt8x"
Oct 30 21:46:00.584231 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:00.584209    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-4lt8x"
Oct 30 21:46:00.617091 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:00.617063    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvbnfd"]
Oct 30 21:46:00.620756 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:00.620735    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvbnfd"
Oct 30 21:46:00.673227 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:00.673198    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Oct 30 21:46:00.673346 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:00.673272    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Oct 30 21:46:00.673346 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:00.673323    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-x4h9p\""
Oct 30 21:46:00.678383 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:00.678358    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvbnfd"]
Oct 30 21:46:00.681355 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:00.681333    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-58dhn"]
Oct 30 21:46:00.689267 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:46:00.689242    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99439df0_9c24_4a03_9dff_e95af6f3f3bd.slice/crio-2e029d24647864ca6d551995ab09cccc000f777078ef06fd69fae01a779c45fa WatchSource:0}: Error finding container 2e029d24647864ca6d551995ab09cccc000f777078ef06fd69fae01a779c45fa: Status 404 returned error can't find the container with id 2e029d24647864ca6d551995ab09cccc000f777078ef06fd69fae01a779c45fa
Oct 30 21:46:00.692207 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:00.692185    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aa3fe244-6da5-41cb-a7b7-6c4d2d17d78b-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvbnfd\" (UID: \"aa3fe244-6da5-41cb-a7b7-6c4d2d17d78b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvbnfd"
Oct 30 21:46:00.692287 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:00.692252    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aa3fe244-6da5-41cb-a7b7-6c4d2d17d78b-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvbnfd\" (UID: \"aa3fe244-6da5-41cb-a7b7-6c4d2d17d78b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvbnfd"
Oct 30 21:46:00.692383 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:00.692365    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lmdp\" (UniqueName: \"kubernetes.io/projected/aa3fe244-6da5-41cb-a7b7-6c4d2d17d78b-kube-api-access-4lmdp\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvbnfd\" (UID: \"aa3fe244-6da5-41cb-a7b7-6c4d2d17d78b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvbnfd"
Oct 30 21:46:00.793047 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:00.792984    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aa3fe244-6da5-41cb-a7b7-6c4d2d17d78b-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvbnfd\" (UID: \"aa3fe244-6da5-41cb-a7b7-6c4d2d17d78b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvbnfd"
Oct 30 21:46:00.793047 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:00.793035    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aa3fe244-6da5-41cb-a7b7-6c4d2d17d78b-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvbnfd\" (UID: \"aa3fe244-6da5-41cb-a7b7-6c4d2d17d78b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvbnfd"
Oct 30 21:46:00.793222 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:00.793065    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lmdp\" (UniqueName: \"kubernetes.io/projected/aa3fe244-6da5-41cb-a7b7-6c4d2d17d78b-kube-api-access-4lmdp\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvbnfd\" (UID: \"aa3fe244-6da5-41cb-a7b7-6c4d2d17d78b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvbnfd"
Oct 30 21:46:00.793429 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:00.793410    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aa3fe244-6da5-41cb-a7b7-6c4d2d17d78b-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvbnfd\" (UID: \"aa3fe244-6da5-41cb-a7b7-6c4d2d17d78b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvbnfd"
Oct 30 21:46:00.793499 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:00.793431    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aa3fe244-6da5-41cb-a7b7-6c4d2d17d78b-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvbnfd\" (UID: \"aa3fe244-6da5-41cb-a7b7-6c4d2d17d78b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvbnfd"
Oct 30 21:46:00.846767 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:00.846745    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lmdp\" (UniqueName: \"kubernetes.io/projected/aa3fe244-6da5-41cb-a7b7-6c4d2d17d78b-kube-api-access-4lmdp\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvbnfd\" (UID: \"aa3fe244-6da5-41cb-a7b7-6c4d2d17d78b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvbnfd"
Oct 30 21:46:00.938761 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:00.938737    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvbnfd"
Oct 30 21:46:00.949654 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:00.949635    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-4lt8x"]
Oct 30 21:46:01.305378 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:01.305346    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvbnfd"]
Oct 30 21:46:01.311612 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:46:01.311585    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa3fe244_6da5_41cb_a7b7_6c4d2d17d78b.slice/crio-d1262209a1599047d1fa906cf9d37785fcbcc651047e1a93097b1be07a706d5e WatchSource:0}: Error finding container d1262209a1599047d1fa906cf9d37785fcbcc651047e1a93097b1be07a706d5e: Status 404 returned error can't find the container with id d1262209a1599047d1fa906cf9d37785fcbcc651047e1a93097b1be07a706d5e
Oct 30 21:46:01.544983 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:01.544950    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-58dhn" event={"ID":"99439df0-9c24-4a03-9dff-e95af6f3f3bd","Type":"ContainerStarted","Data":"2e029d24647864ca6d551995ab09cccc000f777078ef06fd69fae01a779c45fa"}
Oct 30 21:46:01.546093 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:01.546069    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-4lt8x" event={"ID":"36808b98-fda2-40eb-86a0-923a27e268ac","Type":"ContainerStarted","Data":"d09efa032884876d788522f6600978d40deda5dc062491c3d046f471c036623b"}
Oct 30 21:46:01.547136 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:01.547115    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvbnfd" event={"ID":"aa3fe244-6da5-41cb-a7b7-6c4d2d17d78b","Type":"ContainerStarted","Data":"d1262209a1599047d1fa906cf9d37785fcbcc651047e1a93097b1be07a706d5e"}
Oct 30 21:46:05.605924 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:05.605889    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:46:05.606268 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:05.605970    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:46:05.606268 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:05.606011    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:46:05.615104 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:05.615075    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:46:05.615210 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:05.615146    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:46:05.615210 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:05.615192    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:46:06.572513 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:06.572473    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-58dhn" event={"ID":"99439df0-9c24-4a03-9dff-e95af6f3f3bd","Type":"ContainerStarted","Data":"40106f33775b1b36f0a7625562f0efa8af2bc9d98b0fe0f31067b2e8c776b69e"}
Oct 30 21:46:06.574395 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:06.574365    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-4lt8x" event={"ID":"36808b98-fda2-40eb-86a0-923a27e268ac","Type":"ContainerStarted","Data":"3853ff8eb1d6d6489d039aac32d0550c0fe6c9a4008ae954357705b2a2cb0f6a"}
Oct 30 21:46:06.584832 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:06.584806    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-4lt8x"
Oct 30 21:46:06.590271 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:06.590241    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-4lt8x"
Oct 30 21:46:06.732787 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:06.732737    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-58dhn" podStartSLOduration=2.80967288 podStartE2EDuration="7.732723846s" totalImagesPullingTime="4.923050966s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:06.732289095 +0000 UTC m=+220.345058876" watchObservedRunningTime="2026-10-30 21:46:06.732723846 +0000 UTC m=+220.345493604"
Oct 30 21:46:06.846116 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:06.846016    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-4lt8x" podStartSLOduration=2.200394582 podStartE2EDuration="6.846001364s" totalImagesPullingTime="4.645606782s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:46:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:06.845085108 +0000 UTC m=+220.457854886" watchObservedRunningTime="2026-10-30 21:46:06.846001364 +0000 UTC m=+220.458771128"
Oct 30 21:46:07.263753 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:07.263667    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-58dhn"
Oct 30 21:46:07.269340 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:07.269314    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-58dhn"
Oct 30 21:46:07.579763 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:07.579666    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-4lt8x"
Oct 30 21:46:07.579763 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:07.579730    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-58dhn"
Oct 30 21:46:07.581059 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:07.581003    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-4lt8x"
Oct 30 21:46:07.581314 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:07.581296    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-58dhn"
Oct 30 21:46:08.584145 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:08.584111    5061 generic.go:356] "Generic (PLEG): container finished" podID="aa3fe244-6da5-41cb-a7b7-6c4d2d17d78b" containerID="ecf2edd7692a89349e63ce13466dfd4780e86940c3fb4aee8f236665cd75c963" exitCode=0
Oct 30 21:46:08.584566 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:08.584191    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvbnfd" event={"ID":"aa3fe244-6da5-41cb-a7b7-6c4d2d17d78b","Type":"ContainerDied","Data":"ecf2edd7692a89349e63ce13466dfd4780e86940c3fb4aee8f236665cd75c963"}
Oct 30 21:46:10.593719 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:10.593681    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvbnfd" event={"ID":"aa3fe244-6da5-41cb-a7b7-6c4d2d17d78b","Type":"ContainerStarted","Data":"9ab639163096f594625d66b446c608ed3c8da9f152bf62634dc94a5be434f886"}
Oct 30 21:46:11.599764 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:11.599728    5061 generic.go:356] "Generic (PLEG): container finished" podID="aa3fe244-6da5-41cb-a7b7-6c4d2d17d78b" containerID="9ab639163096f594625d66b446c608ed3c8da9f152bf62634dc94a5be434f886" exitCode=0
Oct 30 21:46:11.600214 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:11.599813    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvbnfd" event={"ID":"aa3fe244-6da5-41cb-a7b7-6c4d2d17d78b","Type":"ContainerDied","Data":"9ab639163096f594625d66b446c608ed3c8da9f152bf62634dc94a5be434f886"}
Oct 30 21:46:18.632009 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:18.631975    5061 generic.go:356] "Generic (PLEG): container finished" podID="aa3fe244-6da5-41cb-a7b7-6c4d2d17d78b" containerID="96ac80707ec6c007d14a5ddbde72c352e3969a66b2ee3a4829031f315474e824" exitCode=0
Oct 30 21:46:18.632313 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:18.632011    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvbnfd" event={"ID":"aa3fe244-6da5-41cb-a7b7-6c4d2d17d78b","Type":"ContainerDied","Data":"96ac80707ec6c007d14a5ddbde72c352e3969a66b2ee3a4829031f315474e824"}
Oct 30 21:46:19.762471 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:19.762444    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvbnfd"
Oct 30 21:46:19.863743 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:19.863714    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa3fe244-6da5-41cb-a7b7-6c4d2d17d78b-util\" (UniqueName: \"kubernetes.io/empty-dir/aa3fe244-6da5-41cb-a7b7-6c4d2d17d78b-util\") pod \"aa3fe244-6da5-41cb-a7b7-6c4d2d17d78b\" (UID: \"aa3fe244-6da5-41cb-a7b7-6c4d2d17d78b\") "
Oct 30 21:46:19.863909 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:19.863782    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa3fe244-6da5-41cb-a7b7-6c4d2d17d78b-kube-api-access-4lmdp\" (UniqueName: \"kubernetes.io/projected/aa3fe244-6da5-41cb-a7b7-6c4d2d17d78b-kube-api-access-4lmdp\") pod \"aa3fe244-6da5-41cb-a7b7-6c4d2d17d78b\" (UID: \"aa3fe244-6da5-41cb-a7b7-6c4d2d17d78b\") "
Oct 30 21:46:19.863909 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:19.863824    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa3fe244-6da5-41cb-a7b7-6c4d2d17d78b-bundle\" (UniqueName: \"kubernetes.io/empty-dir/aa3fe244-6da5-41cb-a7b7-6c4d2d17d78b-bundle\") pod \"aa3fe244-6da5-41cb-a7b7-6c4d2d17d78b\" (UID: \"aa3fe244-6da5-41cb-a7b7-6c4d2d17d78b\") "
Oct 30 21:46:19.864367 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:19.864336    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa3fe244-6da5-41cb-a7b7-6c4d2d17d78b-bundle" pod "aa3fe244-6da5-41cb-a7b7-6c4d2d17d78b" (UID: "aa3fe244-6da5-41cb-a7b7-6c4d2d17d78b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:46:19.866029 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:19.866007    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa3fe244-6da5-41cb-a7b7-6c4d2d17d78b-kube-api-access-4lmdp" pod "aa3fe244-6da5-41cb-a7b7-6c4d2d17d78b" (UID: "aa3fe244-6da5-41cb-a7b7-6c4d2d17d78b"). InnerVolumeSpecName "kube-api-access-4lmdp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:46:19.869152 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:19.869127    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa3fe244-6da5-41cb-a7b7-6c4d2d17d78b-util" pod "aa3fe244-6da5-41cb-a7b7-6c4d2d17d78b" (UID: "aa3fe244-6da5-41cb-a7b7-6c4d2d17d78b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:46:19.964513 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:19.964433    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lmdp\" (UniqueName: \"kubernetes.io/projected/aa3fe244-6da5-41cb-a7b7-6c4d2d17d78b-kube-api-access-4lmdp\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:46:19.964513 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:19.964460    5061 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aa3fe244-6da5-41cb-a7b7-6c4d2d17d78b-bundle\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:46:19.964513 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:19.964470    5061 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aa3fe244-6da5-41cb-a7b7-6c4d2d17d78b-util\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:46:20.642684 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:20.642643    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvbnfd" event={"ID":"aa3fe244-6da5-41cb-a7b7-6c4d2d17d78b","Type":"ContainerDied","Data":"d1262209a1599047d1fa906cf9d37785fcbcc651047e1a93097b1be07a706d5e"}
Oct 30 21:46:20.642837 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:20.642711    5061 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d1262209a1599047d1fa906cf9d37785fcbcc651047e1a93097b1be07a706d5e"
Oct 30 21:46:20.642837 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:20.642655    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvbnfd"
Oct 30 21:46:24.665862 ip-10-0-130-60 kubenswrapper[5061]: E1030 21:46:24.665826    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 21:46:42.955406 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:42.955373    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-8677f9fb98-h48ht"]
Oct 30 21:46:42.955905 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:42.955713    5061 cpu_manager.go:414] "containerMap: removing container" podUID="aa3fe244-6da5-41cb-a7b7-6c4d2d17d78b" containerName="pull"
Oct 30 21:46:42.955905 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:42.955725    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa3fe244-6da5-41cb-a7b7-6c4d2d17d78b" containerName="pull"
Oct 30 21:46:42.955905 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:42.955738    5061 cpu_manager.go:414] "containerMap: removing container" podUID="aa3fe244-6da5-41cb-a7b7-6c4d2d17d78b" containerName="extract"
Oct 30 21:46:42.955905 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:42.955744    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa3fe244-6da5-41cb-a7b7-6c4d2d17d78b" containerName="extract"
Oct 30 21:46:42.955905 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:42.955761    5061 cpu_manager.go:414] "containerMap: removing container" podUID="aa3fe244-6da5-41cb-a7b7-6c4d2d17d78b" containerName="util"
Oct 30 21:46:42.955905 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:42.955766    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa3fe244-6da5-41cb-a7b7-6c4d2d17d78b" containerName="util"
Oct 30 21:46:42.955905 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:42.955827    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa3fe244-6da5-41cb-a7b7-6c4d2d17d78b" containerName="extract"
Oct 30 21:46:42.958991 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:42.958953    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8677f9fb98-h48ht"
Oct 30 21:46:43.020769 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:43.020739    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Oct 30 21:46:43.020968 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:43.020809    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Oct 30 21:46:43.054611 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:43.054580    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d9a10237-ed5f-4ba0-a583-48efece29513-webhook-cert\") pod \"opendatahub-operator-controller-manager-8677f9fb98-h48ht\" (UID: \"d9a10237-ed5f-4ba0-a583-48efece29513\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8677f9fb98-h48ht"
Oct 30 21:46:43.054771 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:43.054621    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d9a10237-ed5f-4ba0-a583-48efece29513-manifests\") pod \"opendatahub-operator-controller-manager-8677f9fb98-h48ht\" (UID: \"d9a10237-ed5f-4ba0-a583-48efece29513\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8677f9fb98-h48ht"
Oct 30 21:46:43.054771 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:43.054729    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d9a10237-ed5f-4ba0-a583-48efece29513-metrics-cert\") pod \"opendatahub-operator-controller-manager-8677f9fb98-h48ht\" (UID: \"d9a10237-ed5f-4ba0-a583-48efece29513\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8677f9fb98-h48ht"
Oct 30 21:46:43.054771 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:43.054758    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llfgx\" (UniqueName: \"kubernetes.io/projected/d9a10237-ed5f-4ba0-a583-48efece29513-kube-api-access-llfgx\") pod \"opendatahub-operator-controller-manager-8677f9fb98-h48ht\" (UID: \"d9a10237-ed5f-4ba0-a583-48efece29513\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8677f9fb98-h48ht"
Oct 30 21:46:43.054934 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:43.054797    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d9a10237-ed5f-4ba0-a583-48efece29513-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8677f9fb98-h48ht\" (UID: \"d9a10237-ed5f-4ba0-a583-48efece29513\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8677f9fb98-h48ht"
Oct 30 21:46:43.054934 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:43.054912    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d9a10237-ed5f-4ba0-a583-48efece29513-tmp\") pod \"opendatahub-operator-controller-manager-8677f9fb98-h48ht\" (UID: \"d9a10237-ed5f-4ba0-a583-48efece29513\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8677f9fb98-h48ht"
Oct 30 21:46:43.056365 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:43.056341    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Oct 30 21:46:43.056365 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:43.056344    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-5cc62\""
Oct 30 21:46:43.056520 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:43.056375    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Oct 30 21:46:43.056520 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:43.056381    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Oct 30 21:46:43.058298 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:43.058281    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-8677f9fb98-h48ht"]
Oct 30 21:46:43.155682 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:43.155650    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d9a10237-ed5f-4ba0-a583-48efece29513-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8677f9fb98-h48ht\" (UID: \"d9a10237-ed5f-4ba0-a583-48efece29513\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8677f9fb98-h48ht"
Oct 30 21:46:43.155682 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:43.155686    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d9a10237-ed5f-4ba0-a583-48efece29513-tmp\") pod \"opendatahub-operator-controller-manager-8677f9fb98-h48ht\" (UID: \"d9a10237-ed5f-4ba0-a583-48efece29513\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8677f9fb98-h48ht"
Oct 30 21:46:43.155936 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:43.155719    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d9a10237-ed5f-4ba0-a583-48efece29513-webhook-cert\") pod \"opendatahub-operator-controller-manager-8677f9fb98-h48ht\" (UID: \"d9a10237-ed5f-4ba0-a583-48efece29513\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8677f9fb98-h48ht"
Oct 30 21:46:43.155936 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:43.155738    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d9a10237-ed5f-4ba0-a583-48efece29513-manifests\") pod \"opendatahub-operator-controller-manager-8677f9fb98-h48ht\" (UID: \"d9a10237-ed5f-4ba0-a583-48efece29513\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8677f9fb98-h48ht"
Oct 30 21:46:43.155936 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:43.155856    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d9a10237-ed5f-4ba0-a583-48efece29513-metrics-cert\") pod \"opendatahub-operator-controller-manager-8677f9fb98-h48ht\" (UID: \"d9a10237-ed5f-4ba0-a583-48efece29513\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8677f9fb98-h48ht"
Oct 30 21:46:43.155936 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:43.155889    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llfgx\" (UniqueName: \"kubernetes.io/projected/d9a10237-ed5f-4ba0-a583-48efece29513-kube-api-access-llfgx\") pod \"opendatahub-operator-controller-manager-8677f9fb98-h48ht\" (UID: \"d9a10237-ed5f-4ba0-a583-48efece29513\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8677f9fb98-h48ht"
Oct 30 21:46:43.156238 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:43.156208    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d9a10237-ed5f-4ba0-a583-48efece29513-manifests\") pod \"opendatahub-operator-controller-manager-8677f9fb98-h48ht\" (UID: \"d9a10237-ed5f-4ba0-a583-48efece29513\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8677f9fb98-h48ht"
Oct 30 21:46:43.158297 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:43.158256    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d9a10237-ed5f-4ba0-a583-48efece29513-tmp\") pod \"opendatahub-operator-controller-manager-8677f9fb98-h48ht\" (UID: \"d9a10237-ed5f-4ba0-a583-48efece29513\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8677f9fb98-h48ht"
Oct 30 21:46:43.158396 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:43.158299    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d9a10237-ed5f-4ba0-a583-48efece29513-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8677f9fb98-h48ht\" (UID: \"d9a10237-ed5f-4ba0-a583-48efece29513\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8677f9fb98-h48ht"
Oct 30 21:46:43.158438 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:43.158400    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d9a10237-ed5f-4ba0-a583-48efece29513-webhook-cert\") pod \"opendatahub-operator-controller-manager-8677f9fb98-h48ht\" (UID: \"d9a10237-ed5f-4ba0-a583-48efece29513\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8677f9fb98-h48ht"
Oct 30 21:46:43.158483 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:43.158464    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d9a10237-ed5f-4ba0-a583-48efece29513-metrics-cert\") pod \"opendatahub-operator-controller-manager-8677f9fb98-h48ht\" (UID: \"d9a10237-ed5f-4ba0-a583-48efece29513\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8677f9fb98-h48ht"
Oct 30 21:46:43.211158 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:43.211085    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-llfgx\" (UniqueName: \"kubernetes.io/projected/d9a10237-ed5f-4ba0-a583-48efece29513-kube-api-access-llfgx\") pod \"opendatahub-operator-controller-manager-8677f9fb98-h48ht\" (UID: \"d9a10237-ed5f-4ba0-a583-48efece29513\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8677f9fb98-h48ht"
Oct 30 21:46:43.274036 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:43.274001    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8677f9fb98-h48ht"
Oct 30 21:46:43.807513 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:43.807479    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-8677f9fb98-h48ht"]
Oct 30 21:46:43.813439 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:46:43.813409    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9a10237_ed5f_4ba0_a583_48efece29513.slice/crio-ab65dcfcc44f515af8fcfb4ae856ac693dfc657af9a42e23ea8861cc5ee359ec WatchSource:0}: Error finding container ab65dcfcc44f515af8fcfb4ae856ac693dfc657af9a42e23ea8861cc5ee359ec: Status 404 returned error can't find the container with id ab65dcfcc44f515af8fcfb4ae856ac693dfc657af9a42e23ea8861cc5ee359ec
Oct 30 21:46:44.739140 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:44.739109    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8677f9fb98-h48ht" event={"ID":"d9a10237-ed5f-4ba0-a583-48efece29513","Type":"ContainerStarted","Data":"ab65dcfcc44f515af8fcfb4ae856ac693dfc657af9a42e23ea8861cc5ee359ec"}
Oct 30 21:46:48.758157 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:48.758068    5061 generic.go:356] "Generic (PLEG): container finished" podID="d9a10237-ed5f-4ba0-a583-48efece29513" containerID="6fd0452b661d5e82623ae83a1797be509bc0ae1977c33789e65ec724547ad385" exitCode=0
Oct 30 21:46:48.758573 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:48.758150    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8677f9fb98-h48ht" event={"ID":"d9a10237-ed5f-4ba0-a583-48efece29513","Type":"ContainerDied","Data":"6fd0452b661d5e82623ae83a1797be509bc0ae1977c33789e65ec724547ad385"}
Oct 30 21:46:49.764439 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:49.764403    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8677f9fb98-h48ht" event={"ID":"d9a10237-ed5f-4ba0-a583-48efece29513","Type":"ContainerStarted","Data":"e7faf007c98d43f8c9b98628c0ffa5ebb757e1ee7bc872ded586e26d316f5139"}
Oct 30 21:46:49.764863 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:49.764517    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8677f9fb98-h48ht"
Oct 30 21:46:49.997818 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:46:49.997721    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8677f9fb98-h48ht" podStartSLOduration=2.76458378 podStartE2EDuration="7.997707663s" totalImagesPullingTime="4.848736424s" totalInitContainerRuntime="384.387459ms" isStatefulPod=false podCreationTimestamp="2026-10-30 21:46:42 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:49.949509191 +0000 UTC m=+263.562278981" watchObservedRunningTime="2026-10-30 21:46:49.997707663 +0000 UTC m=+263.610477421"
Oct 30 21:46:54.665824 ip-10-0-130-60 kubenswrapper[5061]: E1030 21:46:54.665784    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 21:47:00.770912 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:00.770885    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8677f9fb98-h48ht"
Oct 30 21:47:05.678413 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:05.678375    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-bm9ls"]
Oct 30 21:47:05.684925 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:05.684902    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-bm9ls"
Oct 30 21:47:05.738399 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:05.738367    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-hfw5l\""
Oct 30 21:47:05.785305 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:05.785270    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Oct 30 21:47:05.785305 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:05.785270    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 30 21:47:05.785586 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:05.785358    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 30 21:47:05.786745 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:05.786719    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-bm9ls"]
Oct 30 21:47:05.858520 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:05.858481    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfwff\" (UniqueName: \"kubernetes.io/projected/902c9b85-ffd2-47c1-abc0-d7a30633a3ad-kube-api-access-qfwff\") pod \"dns-operator-controller-manager-8557cbdb84-bm9ls\" (UID: \"902c9b85-ffd2-47c1-abc0-d7a30633a3ad\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-bm9ls"
Oct 30 21:47:05.959938 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:05.959855    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfwff\" (UniqueName: \"kubernetes.io/projected/902c9b85-ffd2-47c1-abc0-d7a30633a3ad-kube-api-access-qfwff\") pod \"dns-operator-controller-manager-8557cbdb84-bm9ls\" (UID: \"902c9b85-ffd2-47c1-abc0-d7a30633a3ad\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-bm9ls"
Oct 30 21:47:06.015973 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:06.015934    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfwff\" (UniqueName: \"kubernetes.io/projected/902c9b85-ffd2-47c1-abc0-d7a30633a3ad-kube-api-access-qfwff\") pod \"dns-operator-controller-manager-8557cbdb84-bm9ls\" (UID: \"902c9b85-ffd2-47c1-abc0-d7a30633a3ad\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-bm9ls"
Oct 30 21:47:06.297571 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:06.297516    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-bm9ls"
Oct 30 21:47:06.727728 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:06.727646    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-bm9ls"]
Oct 30 21:47:06.733764 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:47:06.733738    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod902c9b85_ffd2_47c1_abc0_d7a30633a3ad.slice/crio-97818cc2a706d6e21005714d5f1360836ecf69c02c5d32430cefc88eabbf3397 WatchSource:0}: Error finding container 97818cc2a706d6e21005714d5f1360836ecf69c02c5d32430cefc88eabbf3397: Status 404 returned error can't find the container with id 97818cc2a706d6e21005714d5f1360836ecf69c02c5d32430cefc88eabbf3397
Oct 30 21:47:06.840199 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:06.840158    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-bm9ls" event={"ID":"902c9b85-ffd2-47c1-abc0-d7a30633a3ad","Type":"ContainerStarted","Data":"97818cc2a706d6e21005714d5f1360836ecf69c02c5d32430cefc88eabbf3397"}
Oct 30 21:47:09.952170 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:09.952132    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-l2gwp"]
Oct 30 21:47:09.963944 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:09.963920    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-l2gwp"
Oct 30 21:47:10.013785 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:10.013759    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-w8n9c\""
Oct 30 21:47:10.062566 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:10.062520    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-l2gwp"]
Oct 30 21:47:10.096214 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:10.096177    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4rsz\" (UniqueName: \"kubernetes.io/projected/8e0f4d14-ce8b-42a3-8b8e-39db40615683-kube-api-access-r4rsz\") pod \"authorino-operator-7b7bb4bc77-l2gwp\" (UID: \"8e0f4d14-ce8b-42a3-8b8e-39db40615683\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-l2gwp"
Oct 30 21:47:10.197195 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:10.197161    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4rsz\" (UniqueName: \"kubernetes.io/projected/8e0f4d14-ce8b-42a3-8b8e-39db40615683-kube-api-access-r4rsz\") pod \"authorino-operator-7b7bb4bc77-l2gwp\" (UID: \"8e0f4d14-ce8b-42a3-8b8e-39db40615683\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-l2gwp"
Oct 30 21:47:10.258692 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:10.258519    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4rsz\" (UniqueName: \"kubernetes.io/projected/8e0f4d14-ce8b-42a3-8b8e-39db40615683-kube-api-access-r4rsz\") pod \"authorino-operator-7b7bb4bc77-l2gwp\" (UID: \"8e0f4d14-ce8b-42a3-8b8e-39db40615683\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-l2gwp"
Oct 30 21:47:10.280004 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:10.279972    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-l2gwp"
Oct 30 21:47:10.662053 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:10.662021    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-l2gwp"]
Oct 30 21:47:10.668109 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:47:10.668080    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e0f4d14_ce8b_42a3_8b8e_39db40615683.slice/crio-88c112ae201dcf973cfe6c5661dd146eb647b505bf19646f8726df7654ac898c WatchSource:0}: Error finding container 88c112ae201dcf973cfe6c5661dd146eb647b505bf19646f8726df7654ac898c: Status 404 returned error can't find the container with id 88c112ae201dcf973cfe6c5661dd146eb647b505bf19646f8726df7654ac898c
Oct 30 21:47:10.860179 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:10.860141    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-l2gwp" event={"ID":"8e0f4d14-ce8b-42a3-8b8e-39db40615683","Type":"ContainerStarted","Data":"88c112ae201dcf973cfe6c5661dd146eb647b505bf19646f8726df7654ac898c"}
Oct 30 21:47:10.861713 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:10.861689    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-bm9ls" event={"ID":"902c9b85-ffd2-47c1-abc0-d7a30633a3ad","Type":"ContainerStarted","Data":"87fc918c7b77b9568d278a36bdb15f3bcc9b18f7b5a246b6b9277c4104f6a411"}
Oct 30 21:47:10.861823 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:10.861813    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-bm9ls"
Oct 30 21:47:10.979839 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:10.979753    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-bm9ls" podStartSLOduration=2.272250234 podStartE2EDuration="5.979740581s" totalImagesPullingTime="3.707490347s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:47:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:10.97963945 +0000 UTC m=+284.592409239" watchObservedRunningTime="2026-10-30 21:47:10.979740581 +0000 UTC m=+284.592510316"
Oct 30 21:47:13.877987 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:13.877947    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-l2gwp" event={"ID":"8e0f4d14-ce8b-42a3-8b8e-39db40615683","Type":"ContainerStarted","Data":"43a5fa24ee41f4a4ad63d55f19f933881241eda0c1ad0c6bba92d6969bb0a5db"}
Oct 30 21:47:13.878432 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:13.878075    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-l2gwp"
Oct 30 21:47:14.002802 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:14.002747    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-l2gwp" podStartSLOduration=2.629389335 podStartE2EDuration="5.002733978s" totalImagesPullingTime="2.373344643s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:47:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:14.001841045 +0000 UTC m=+287.614610804" watchObservedRunningTime="2026-10-30 21:47:14.002733978 +0000 UTC m=+287.615503735"
Oct 30 21:47:14.080158 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:14.080125    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-j77xb"]
Oct 30 21:47:14.083794 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:14.083773    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-j77xb"
Oct 30 21:47:14.133367 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:14.133302    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2smg\" (UniqueName: \"kubernetes.io/projected/743ad69b-c9fa-4fb5-b942-baa0c949a7af-kube-api-access-x2smg\") pod \"limitador-operator-controller-manager-5778bb4449-j77xb\" (UID: \"743ad69b-c9fa-4fb5-b942-baa0c949a7af\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-j77xb"
Oct 30 21:47:14.146279 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:14.146254    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-j77xb"]
Oct 30 21:47:14.146402 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:14.146382    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-w2xg7\""
Oct 30 21:47:14.233875 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:14.233837    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2smg\" (UniqueName: \"kubernetes.io/projected/743ad69b-c9fa-4fb5-b942-baa0c949a7af-kube-api-access-x2smg\") pod \"limitador-operator-controller-manager-5778bb4449-j77xb\" (UID: \"743ad69b-c9fa-4fb5-b942-baa0c949a7af\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-j77xb"
Oct 30 21:47:14.291788 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:14.291756    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2smg\" (UniqueName: \"kubernetes.io/projected/743ad69b-c9fa-4fb5-b942-baa0c949a7af-kube-api-access-x2smg\") pod \"limitador-operator-controller-manager-5778bb4449-j77xb\" (UID: \"743ad69b-c9fa-4fb5-b942-baa0c949a7af\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-j77xb"
Oct 30 21:47:14.395937 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:14.395855    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-j77xb"
Oct 30 21:47:14.767862 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:14.767784    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-j77xb"]
Oct 30 21:47:14.773501 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:47:14.773471    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod743ad69b_c9fa_4fb5_b942_baa0c949a7af.slice/crio-1ed1356ea9b14d658630230043aef08e6f884f46071e49648b481189aaf4ed4e WatchSource:0}: Error finding container 1ed1356ea9b14d658630230043aef08e6f884f46071e49648b481189aaf4ed4e: Status 404 returned error can't find the container with id 1ed1356ea9b14d658630230043aef08e6f884f46071e49648b481189aaf4ed4e
Oct 30 21:47:14.883878 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:14.883843    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-j77xb" event={"ID":"743ad69b-c9fa-4fb5-b942-baa0c949a7af","Type":"ContainerStarted","Data":"1ed1356ea9b14d658630230043aef08e6f884f46071e49648b481189aaf4ed4e"}
Oct 30 21:47:17.208448 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:17.208416    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-25bmv"]
Oct 30 21:47:17.212694 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:17.212666    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-25bmv"
Oct 30 21:47:17.257372 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:17.257333    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fctx\" (UniqueName: \"kubernetes.io/projected/db9fd2b8-dcbf-4145-9d53-6f030812a88d-kube-api-access-5fctx\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-25bmv\" (UID: \"db9fd2b8-dcbf-4145-9d53-6f030812a88d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-25bmv"
Oct 30 21:47:17.257559 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:17.257384    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db9fd2b8-dcbf-4145-9d53-6f030812a88d-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-25bmv\" (UID: \"db9fd2b8-dcbf-4145-9d53-6f030812a88d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-25bmv"
Oct 30 21:47:17.257559 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:17.257407    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db9fd2b8-dcbf-4145-9d53-6f030812a88d-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-25bmv\" (UID: \"db9fd2b8-dcbf-4145-9d53-6f030812a88d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-25bmv"
Oct 30 21:47:17.257559 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:17.257455    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db9fd2b8-dcbf-4145-9d53-6f030812a88d-istio-token\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-25bmv\" (UID: \"db9fd2b8-dcbf-4145-9d53-6f030812a88d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-25bmv"
Oct 30 21:47:17.257559 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:17.257482    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db9fd2b8-dcbf-4145-9d53-6f030812a88d-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-25bmv\" (UID: \"db9fd2b8-dcbf-4145-9d53-6f030812a88d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-25bmv"
Oct 30 21:47:17.257729 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:17.257568    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db9fd2b8-dcbf-4145-9d53-6f030812a88d-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-25bmv\" (UID: \"db9fd2b8-dcbf-4145-9d53-6f030812a88d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-25bmv"
Oct 30 21:47:17.257729 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:17.257613    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db9fd2b8-dcbf-4145-9d53-6f030812a88d-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-25bmv\" (UID: \"db9fd2b8-dcbf-4145-9d53-6f030812a88d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-25bmv"
Oct 30 21:47:17.257729 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:17.257647    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db9fd2b8-dcbf-4145-9d53-6f030812a88d-istio-data\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-25bmv\" (UID: \"db9fd2b8-dcbf-4145-9d53-6f030812a88d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-25bmv"
Oct 30 21:47:17.257729 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:17.257672    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db9fd2b8-dcbf-4145-9d53-6f030812a88d-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-25bmv\" (UID: \"db9fd2b8-dcbf-4145-9d53-6f030812a88d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-25bmv"
Oct 30 21:47:17.265946 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:17.265917    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-89cjw\""
Oct 30 21:47:17.314307 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:17.314226    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-25bmv"]
Oct 30 21:47:17.358321 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:17.358288    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db9fd2b8-dcbf-4145-9d53-6f030812a88d-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-25bmv\" (UID: \"db9fd2b8-dcbf-4145-9d53-6f030812a88d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-25bmv"
Oct 30 21:47:17.358488 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:17.358333    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db9fd2b8-dcbf-4145-9d53-6f030812a88d-istio-data\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-25bmv\" (UID: \"db9fd2b8-dcbf-4145-9d53-6f030812a88d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-25bmv"
Oct 30 21:47:17.358488 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:17.358465    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db9fd2b8-dcbf-4145-9d53-6f030812a88d-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-25bmv\" (UID: \"db9fd2b8-dcbf-4145-9d53-6f030812a88d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-25bmv"
Oct 30 21:47:17.358637 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:17.358542    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fctx\" (UniqueName: \"kubernetes.io/projected/db9fd2b8-dcbf-4145-9d53-6f030812a88d-kube-api-access-5fctx\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-25bmv\" (UID: \"db9fd2b8-dcbf-4145-9d53-6f030812a88d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-25bmv"
Oct 30 21:47:17.358637 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:17.358584    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db9fd2b8-dcbf-4145-9d53-6f030812a88d-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-25bmv\" (UID: \"db9fd2b8-dcbf-4145-9d53-6f030812a88d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-25bmv"
Oct 30 21:47:17.358637 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:17.358619    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db9fd2b8-dcbf-4145-9d53-6f030812a88d-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-25bmv\" (UID: \"db9fd2b8-dcbf-4145-9d53-6f030812a88d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-25bmv"
Oct 30 21:47:17.358789 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:17.358639    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db9fd2b8-dcbf-4145-9d53-6f030812a88d-istio-data\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-25bmv\" (UID: \"db9fd2b8-dcbf-4145-9d53-6f030812a88d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-25bmv"
Oct 30 21:47:17.358789 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:17.358697    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db9fd2b8-dcbf-4145-9d53-6f030812a88d-istio-token\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-25bmv\" (UID: \"db9fd2b8-dcbf-4145-9d53-6f030812a88d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-25bmv"
Oct 30 21:47:17.358789 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:17.358728    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db9fd2b8-dcbf-4145-9d53-6f030812a88d-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-25bmv\" (UID: \"db9fd2b8-dcbf-4145-9d53-6f030812a88d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-25bmv"
Oct 30 21:47:17.358789 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:17.358745    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db9fd2b8-dcbf-4145-9d53-6f030812a88d-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-25bmv\" (UID: \"db9fd2b8-dcbf-4145-9d53-6f030812a88d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-25bmv"
Oct 30 21:47:17.358789 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:17.358770    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db9fd2b8-dcbf-4145-9d53-6f030812a88d-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-25bmv\" (UID: \"db9fd2b8-dcbf-4145-9d53-6f030812a88d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-25bmv"
Oct 30 21:47:17.359026 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:17.358809    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db9fd2b8-dcbf-4145-9d53-6f030812a88d-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-25bmv\" (UID: \"db9fd2b8-dcbf-4145-9d53-6f030812a88d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-25bmv"
Oct 30 21:47:17.359151 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:17.359124    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db9fd2b8-dcbf-4145-9d53-6f030812a88d-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-25bmv\" (UID: \"db9fd2b8-dcbf-4145-9d53-6f030812a88d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-25bmv"
Oct 30 21:47:17.359359 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:17.359338    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db9fd2b8-dcbf-4145-9d53-6f030812a88d-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-25bmv\" (UID: \"db9fd2b8-dcbf-4145-9d53-6f030812a88d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-25bmv"
Oct 30 21:47:17.360738 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:17.360718    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db9fd2b8-dcbf-4145-9d53-6f030812a88d-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-25bmv\" (UID: \"db9fd2b8-dcbf-4145-9d53-6f030812a88d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-25bmv"
Oct 30 21:47:17.361144 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:17.361121    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db9fd2b8-dcbf-4145-9d53-6f030812a88d-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-25bmv\" (UID: \"db9fd2b8-dcbf-4145-9d53-6f030812a88d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-25bmv"
Oct 30 21:47:17.418310 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:17.418279    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fctx\" (UniqueName: \"kubernetes.io/projected/db9fd2b8-dcbf-4145-9d53-6f030812a88d-kube-api-access-5fctx\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-25bmv\" (UID: \"db9fd2b8-dcbf-4145-9d53-6f030812a88d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-25bmv"
Oct 30 21:47:17.419812 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:17.419790    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db9fd2b8-dcbf-4145-9d53-6f030812a88d-istio-token\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-25bmv\" (UID: \"db9fd2b8-dcbf-4145-9d53-6f030812a88d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-25bmv"
Oct 30 21:47:17.527792 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:17.527759    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-25bmv"
Oct 30 21:47:17.899607 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:17.899570    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-j77xb" event={"ID":"743ad69b-c9fa-4fb5-b942-baa0c949a7af","Type":"ContainerStarted","Data":"087787c0df442240eb1a12a706cfedd7acb65855bb93f53c75bb9c07c38bba0f"}
Oct 30 21:47:17.899782 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:17.899699    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-j77xb"
Oct 30 21:47:17.958417 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:17.958388    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-25bmv"]
Oct 30 21:47:17.964082 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:47:17.964056    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb9fd2b8_dcbf_4145_9d53_6f030812a88d.slice/crio-91fe54bcb5c50d9982d0263249a40c9082bfd799fbae8c7401fbee7723f29eed WatchSource:0}: Error finding container 91fe54bcb5c50d9982d0263249a40c9082bfd799fbae8c7401fbee7723f29eed: Status 404 returned error can't find the container with id 91fe54bcb5c50d9982d0263249a40c9082bfd799fbae8c7401fbee7723f29eed
Oct 30 21:47:17.966126 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:17.966098    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:47:17.966225 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:17.966170    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:47:17.966291 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:17.966211    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:47:18.012715 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:18.012671    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-j77xb" podStartSLOduration=1.367461941 podStartE2EDuration="4.012656191s" totalImagesPullingTime="2.64519425s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:47:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:18.012288757 +0000 UTC m=+291.625058506" watchObservedRunningTime="2026-10-30 21:47:18.012656191 +0000 UTC m=+291.625425950"
Oct 30 21:47:18.905816 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:18.905772    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-25bmv" event={"ID":"db9fd2b8-dcbf-4145-9d53-6f030812a88d","Type":"ContainerStarted","Data":"c3014fe0ac6551dd4cf25612f60ab3a30a091b6b2fb900127e222d07191a9f94"}
Oct 30 21:47:18.906212 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:18.905834    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-25bmv" event={"ID":"db9fd2b8-dcbf-4145-9d53-6f030812a88d","Type":"ContainerStarted","Data":"91fe54bcb5c50d9982d0263249a40c9082bfd799fbae8c7401fbee7723f29eed"}
Oct 30 21:47:19.028858 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:19.028802    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-25bmv" podStartSLOduration=2.028787233 podStartE2EDuration="2.028787233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:47:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:19.027861567 +0000 UTC m=+292.640631327" watchObservedRunningTime="2026-10-30 21:47:19.028787233 +0000 UTC m=+292.641556983"
Oct 30 21:47:19.527916 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:19.527883    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-25bmv"
Oct 30 21:47:19.532454 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:19.532428    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-25bmv"
Oct 30 21:47:19.910419 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:19.910387    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-25bmv"
Oct 30 21:47:19.911450 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:19.911434    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-25bmv"
Oct 30 21:47:20.177419 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:20.177340    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-5jdb4"]
Oct 30 21:47:20.181053 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:20.181032    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-5jdb4"
Oct 30 21:47:20.239282 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:20.239263    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-pdssd\""
Oct 30 21:47:20.252575 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:20.252551    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-5jdb4"]
Oct 30 21:47:20.284832 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:20.284803    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwqbs\" (UniqueName: \"kubernetes.io/projected/32db9323-1a34-4407-9d58-9178bd288a9e-kube-api-access-xwqbs\") pod \"authorino-57f7f8659f-5jdb4\" (UID: \"32db9323-1a34-4407-9d58-9178bd288a9e\") " pod="kuadrant-system/authorino-57f7f8659f-5jdb4"
Oct 30 21:47:20.385795 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:20.385763    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwqbs\" (UniqueName: \"kubernetes.io/projected/32db9323-1a34-4407-9d58-9178bd288a9e-kube-api-access-xwqbs\") pod \"authorino-57f7f8659f-5jdb4\" (UID: \"32db9323-1a34-4407-9d58-9178bd288a9e\") " pod="kuadrant-system/authorino-57f7f8659f-5jdb4"
Oct 30 21:47:20.444660 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:20.444600    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwqbs\" (UniqueName: \"kubernetes.io/projected/32db9323-1a34-4407-9d58-9178bd288a9e-kube-api-access-xwqbs\") pod \"authorino-57f7f8659f-5jdb4\" (UID: \"32db9323-1a34-4407-9d58-9178bd288a9e\") " pod="kuadrant-system/authorino-57f7f8659f-5jdb4"
Oct 30 21:47:20.504505 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:20.504476    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-5jdb4"
Oct 30 21:47:20.875116 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:20.875081    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-5jdb4"]
Oct 30 21:47:20.881059 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:47:20.881032    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod32db9323_1a34_4407_9d58_9178bd288a9e.slice/crio-b359318b5f41dc571827f14dec12f45379e92d0693064637ea2127f7dd3c68e0 WatchSource:0}: Error finding container b359318b5f41dc571827f14dec12f45379e92d0693064637ea2127f7dd3c68e0: Status 404 returned error can't find the container with id b359318b5f41dc571827f14dec12f45379e92d0693064637ea2127f7dd3c68e0
Oct 30 21:47:20.915341 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:20.915306    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-5jdb4" event={"ID":"32db9323-1a34-4407-9d58-9178bd288a9e","Type":"ContainerStarted","Data":"b359318b5f41dc571827f14dec12f45379e92d0693064637ea2127f7dd3c68e0"}
Oct 30 21:47:21.867921 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:21.867883    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-bm9ls"
Oct 30 21:47:23.929875 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:23.929841    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-5jdb4" event={"ID":"32db9323-1a34-4407-9d58-9178bd288a9e","Type":"ContainerStarted","Data":"5a630af622deaf51fd544240e6951e6b35ac16b942c6967f1588ef9b578be88f"}
Oct 30 21:47:24.041312 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:24.041201    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-5jdb4" podStartSLOduration=1.14787141 podStartE2EDuration="4.0411852s" totalImagesPullingTime="2.89331379s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:47:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:24.040156662 +0000 UTC m=+297.652926420" watchObservedRunningTime="2026-10-30 21:47:24.0411852 +0000 UTC m=+297.653954957"
Oct 30 21:47:24.666360 ip-10-0-130-60 kubenswrapper[5061]: E1030 21:47:24.666323    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 21:47:24.886510 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:24.886480    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-l2gwp"
Oct 30 21:47:26.549774 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:26.549749    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-sclrt_5e74071e-0462-42b5-a05c-d90bb638478a/console-operator/2.log"
Oct 30 21:47:26.550980 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:26.550954    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-sclrt_5e74071e-0462-42b5-a05c-d90bb638478a/console-operator/2.log"
Oct 30 21:47:28.908320 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:28.908285    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-j77xb"
Oct 30 21:47:31.342112 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:31.342079    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-66dd8b7b5b-lr4tm"]
Oct 30 21:47:31.347143 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:31.347121    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-66dd8b7b5b-lr4tm"
Oct 30 21:47:31.382187 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:31.382155    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/03551684-8f83-405c-8df9-24ea27a7717a-manager-config\") pod \"jobset-controller-manager-66dd8b7b5b-lr4tm\" (UID: \"03551684-8f83-405c-8df9-24ea27a7717a\") " pod="openshift-jobset-operator/jobset-controller-manager-66dd8b7b5b-lr4tm"
Oct 30 21:47:31.382344 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:31.382197    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdl8c\" (UniqueName: \"kubernetes.io/projected/03551684-8f83-405c-8df9-24ea27a7717a-kube-api-access-hdl8c\") pod \"jobset-controller-manager-66dd8b7b5b-lr4tm\" (UID: \"03551684-8f83-405c-8df9-24ea27a7717a\") " pod="openshift-jobset-operator/jobset-controller-manager-66dd8b7b5b-lr4tm"
Oct 30 21:47:31.382344 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:31.382292    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03551684-8f83-405c-8df9-24ea27a7717a-metrics-certs\") pod \"jobset-controller-manager-66dd8b7b5b-lr4tm\" (UID: \"03551684-8f83-405c-8df9-24ea27a7717a\") " pod="openshift-jobset-operator/jobset-controller-manager-66dd8b7b5b-lr4tm"
Oct 30 21:47:31.382421 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:31.382344    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/03551684-8f83-405c-8df9-24ea27a7717a-cert\") pod \"jobset-controller-manager-66dd8b7b5b-lr4tm\" (UID: \"03551684-8f83-405c-8df9-24ea27a7717a\") " pod="openshift-jobset-operator/jobset-controller-manager-66dd8b7b5b-lr4tm"
Oct 30 21:47:31.444880 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:31.444852    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\""
Oct 30 21:47:31.445053 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:31.444924    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\""
Oct 30 21:47:31.445053 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:31.444925    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\""
Oct 30 21:47:31.445053 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:31.444927    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\""
Oct 30 21:47:31.445053 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:31.444934    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\""
Oct 30 21:47:31.445053 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:31.444998    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-fl2lh\""
Oct 30 21:47:31.483492 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:31.483458    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/03551684-8f83-405c-8df9-24ea27a7717a-manager-config\") pod \"jobset-controller-manager-66dd8b7b5b-lr4tm\" (UID: \"03551684-8f83-405c-8df9-24ea27a7717a\") " pod="openshift-jobset-operator/jobset-controller-manager-66dd8b7b5b-lr4tm"
Oct 30 21:47:31.483492 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:31.483497    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdl8c\" (UniqueName: \"kubernetes.io/projected/03551684-8f83-405c-8df9-24ea27a7717a-kube-api-access-hdl8c\") pod \"jobset-controller-manager-66dd8b7b5b-lr4tm\" (UID: \"03551684-8f83-405c-8df9-24ea27a7717a\") " pod="openshift-jobset-operator/jobset-controller-manager-66dd8b7b5b-lr4tm"
Oct 30 21:47:31.483715 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:31.483548    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03551684-8f83-405c-8df9-24ea27a7717a-metrics-certs\") pod \"jobset-controller-manager-66dd8b7b5b-lr4tm\" (UID: \"03551684-8f83-405c-8df9-24ea27a7717a\") " pod="openshift-jobset-operator/jobset-controller-manager-66dd8b7b5b-lr4tm"
Oct 30 21:47:31.483715 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:31.483581    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/03551684-8f83-405c-8df9-24ea27a7717a-cert\") pod \"jobset-controller-manager-66dd8b7b5b-lr4tm\" (UID: \"03551684-8f83-405c-8df9-24ea27a7717a\") " pod="openshift-jobset-operator/jobset-controller-manager-66dd8b7b5b-lr4tm"
Oct 30 21:47:31.484249 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:31.484227    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/03551684-8f83-405c-8df9-24ea27a7717a-manager-config\") pod \"jobset-controller-manager-66dd8b7b5b-lr4tm\" (UID: \"03551684-8f83-405c-8df9-24ea27a7717a\") " pod="openshift-jobset-operator/jobset-controller-manager-66dd8b7b5b-lr4tm"
Oct 30 21:47:31.486096 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:31.486075    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03551684-8f83-405c-8df9-24ea27a7717a-metrics-certs\") pod \"jobset-controller-manager-66dd8b7b5b-lr4tm\" (UID: \"03551684-8f83-405c-8df9-24ea27a7717a\") " pod="openshift-jobset-operator/jobset-controller-manager-66dd8b7b5b-lr4tm"
Oct 30 21:47:31.486187 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:31.486117    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/03551684-8f83-405c-8df9-24ea27a7717a-cert\") pod \"jobset-controller-manager-66dd8b7b5b-lr4tm\" (UID: \"03551684-8f83-405c-8df9-24ea27a7717a\") " pod="openshift-jobset-operator/jobset-controller-manager-66dd8b7b5b-lr4tm"
Oct 30 21:47:31.494060 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:31.494037    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-66dd8b7b5b-lr4tm"]
Oct 30 21:47:31.538574 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:31.538540    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdl8c\" (UniqueName: \"kubernetes.io/projected/03551684-8f83-405c-8df9-24ea27a7717a-kube-api-access-hdl8c\") pod \"jobset-controller-manager-66dd8b7b5b-lr4tm\" (UID: \"03551684-8f83-405c-8df9-24ea27a7717a\") " pod="openshift-jobset-operator/jobset-controller-manager-66dd8b7b5b-lr4tm"
Oct 30 21:47:31.659293 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:31.659215    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-66dd8b7b5b-lr4tm"
Oct 30 21:47:32.028841 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:32.028756    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-66dd8b7b5b-lr4tm"]
Oct 30 21:47:32.034149 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:47:32.034125    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03551684_8f83_405c_8df9_24ea27a7717a.slice/crio-9c07ef107318ffe22c4c3c033e1af797a4f674951d3381b2ed9f2aae6f07b344 WatchSource:0}: Error finding container 9c07ef107318ffe22c4c3c033e1af797a4f674951d3381b2ed9f2aae6f07b344: Status 404 returned error can't find the container with id 9c07ef107318ffe22c4c3c033e1af797a4f674951d3381b2ed9f2aae6f07b344
Oct 30 21:47:32.036083 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:32.036061    5061 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 21:47:32.976004 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:32.975965    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-66dd8b7b5b-lr4tm" event={"ID":"03551684-8f83-405c-8df9-24ea27a7717a","Type":"ContainerStarted","Data":"9c07ef107318ffe22c4c3c033e1af797a4f674951d3381b2ed9f2aae6f07b344"}
Oct 30 21:47:37.002710 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:37.002669    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-66dd8b7b5b-lr4tm" event={"ID":"03551684-8f83-405c-8df9-24ea27a7717a","Type":"ContainerStarted","Data":"16ede609a8c9dc827b81a978ff06b92758262c3746dd6d83f6287a1d791d9284"}
Oct 30 21:47:37.003179 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:37.002802    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-66dd8b7b5b-lr4tm"
Oct 30 21:47:48.012917 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:48.012840    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-66dd8b7b5b-lr4tm"
Oct 30 21:47:48.125891 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:48.125841    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-66dd8b7b5b-lr4tm" podStartSLOduration=13.140583253 podStartE2EDuration="17.125824265s" totalImagesPullingTime="3.985241012s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:47:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:37.119456955 +0000 UTC m=+310.732226712" watchObservedRunningTime="2026-10-30 21:47:48.125824265 +0000 UTC m=+321.738594024"
Oct 30 21:47:54.665783 ip-10-0-130-60 kubenswrapper[5061]: E1030 21:47:54.665747    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 21:47:55.817662 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:55.817626    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-px4hr"]
Oct 30 21:47:55.826439 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:55.826311    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-px4hr"
Oct 30 21:47:55.887657 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:55.887627    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-px4hr"]
Oct 30 21:47:56.002254 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:56.002213    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h85sl\" (UniqueName: \"kubernetes.io/projected/3011467a-326a-4acb-9501-9175ed4cab27-kube-api-access-h85sl\") pod \"authorino-56c6c9ddd5-px4hr\" (UID: \"3011467a-326a-4acb-9501-9175ed4cab27\") " pod="kuadrant-system/authorino-56c6c9ddd5-px4hr"
Oct 30 21:47:56.103672 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:56.103592    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h85sl\" (UniqueName: \"kubernetes.io/projected/3011467a-326a-4acb-9501-9175ed4cab27-kube-api-access-h85sl\") pod \"authorino-56c6c9ddd5-px4hr\" (UID: \"3011467a-326a-4acb-9501-9175ed4cab27\") " pod="kuadrant-system/authorino-56c6c9ddd5-px4hr"
Oct 30 21:47:56.160824 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:56.160794    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h85sl\" (UniqueName: \"kubernetes.io/projected/3011467a-326a-4acb-9501-9175ed4cab27-kube-api-access-h85sl\") pod \"authorino-56c6c9ddd5-px4hr\" (UID: \"3011467a-326a-4acb-9501-9175ed4cab27\") " pod="kuadrant-system/authorino-56c6c9ddd5-px4hr"
Oct 30 21:47:56.438559 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:56.438463    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-px4hr"
Oct 30 21:47:56.801028 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:56.801003    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-px4hr"]
Oct 30 21:47:56.807218 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:47:56.807190    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3011467a_326a_4acb_9501_9175ed4cab27.slice/crio-9120ef374bf38958b2985109b6a87794258416f05ca7450ad96c63e1b773f08d WatchSource:0}: Error finding container 9120ef374bf38958b2985109b6a87794258416f05ca7450ad96c63e1b773f08d: Status 404 returned error can't find the container with id 9120ef374bf38958b2985109b6a87794258416f05ca7450ad96c63e1b773f08d
Oct 30 21:47:57.091515 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:57.091433    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-px4hr" event={"ID":"3011467a-326a-4acb-9501-9175ed4cab27","Type":"ContainerStarted","Data":"9120ef374bf38958b2985109b6a87794258416f05ca7450ad96c63e1b773f08d"}
Oct 30 21:47:59.101586 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:59.101548    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-px4hr" event={"ID":"3011467a-326a-4acb-9501-9175ed4cab27","Type":"ContainerStarted","Data":"3a1d377717328f4fcb8784632cdee0673818f9aae3e366c4c3a874ad1eedf8fd"}
Oct 30 21:47:59.217305 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:59.217254    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-px4hr" podStartSLOduration=2.873702045 podStartE2EDuration="4.217238901s" totalImagesPullingTime="1.343536856s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:47:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:59.215748897 +0000 UTC m=+332.828518669" watchObservedRunningTime="2026-10-30 21:47:59.217238901 +0000 UTC m=+332.830008653"
Oct 30 21:47:59.239236 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:59.239198    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-5jdb4"]
Oct 30 21:47:59.239432 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:59.239408    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-5jdb4" podUID="32db9323-1a34-4407-9d58-9178bd288a9e" containerName="authorino" containerID="cri-o://5a630af622deaf51fd544240e6951e6b35ac16b942c6967f1588ef9b578be88f" gracePeriod=30
Oct 30 21:47:59.483667 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:59.483642    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-5jdb4"
Oct 30 21:47:59.634450 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:59.634378    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32db9323-1a34-4407-9d58-9178bd288a9e-kube-api-access-xwqbs\" (UniqueName: \"kubernetes.io/projected/32db9323-1a34-4407-9d58-9178bd288a9e-kube-api-access-xwqbs\") pod \"32db9323-1a34-4407-9d58-9178bd288a9e\" (UID: \"32db9323-1a34-4407-9d58-9178bd288a9e\") "
Oct 30 21:47:59.636549 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:59.636513    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32db9323-1a34-4407-9d58-9178bd288a9e-kube-api-access-xwqbs" pod "32db9323-1a34-4407-9d58-9178bd288a9e" (UID: "32db9323-1a34-4407-9d58-9178bd288a9e"). InnerVolumeSpecName "kube-api-access-xwqbs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:47:59.735864 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:47:59.735835    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwqbs\" (UniqueName: \"kubernetes.io/projected/32db9323-1a34-4407-9d58-9178bd288a9e-kube-api-access-xwqbs\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:48:00.106832 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:48:00.106799    5061 generic.go:356] "Generic (PLEG): container finished" podID="32db9323-1a34-4407-9d58-9178bd288a9e" containerID="5a630af622deaf51fd544240e6951e6b35ac16b942c6967f1588ef9b578be88f" exitCode=0
Oct 30 21:48:00.107341 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:48:00.106840    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-5jdb4"
Oct 30 21:48:00.107341 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:48:00.106890    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-5jdb4" event={"ID":"32db9323-1a34-4407-9d58-9178bd288a9e","Type":"ContainerDied","Data":"5a630af622deaf51fd544240e6951e6b35ac16b942c6967f1588ef9b578be88f"}
Oct 30 21:48:00.107341 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:48:00.106928    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-5jdb4" event={"ID":"32db9323-1a34-4407-9d58-9178bd288a9e","Type":"ContainerDied","Data":"b359318b5f41dc571827f14dec12f45379e92d0693064637ea2127f7dd3c68e0"}
Oct 30 21:48:00.107341 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:48:00.106948    5061 scope.go:122] "RemoveContainer" containerID="5a630af622deaf51fd544240e6951e6b35ac16b942c6967f1588ef9b578be88f"
Oct 30 21:48:00.118647 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:48:00.118611    5061 scope.go:122] "RemoveContainer" containerID="5a630af622deaf51fd544240e6951e6b35ac16b942c6967f1588ef9b578be88f"
Oct 30 21:48:00.118926 ip-10-0-130-60 kubenswrapper[5061]: E1030 21:48:00.118901    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a630af622deaf51fd544240e6951e6b35ac16b942c6967f1588ef9b578be88f\": container with ID starting with 5a630af622deaf51fd544240e6951e6b35ac16b942c6967f1588ef9b578be88f not found: ID does not exist" containerID="5a630af622deaf51fd544240e6951e6b35ac16b942c6967f1588ef9b578be88f"
Oct 30 21:48:00.118985 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:48:00.118936    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a630af622deaf51fd544240e6951e6b35ac16b942c6967f1588ef9b578be88f"} err="failed to get container status \"5a630af622deaf51fd544240e6951e6b35ac16b942c6967f1588ef9b578be88f\": rpc error: code = NotFound desc = could not find container \"5a630af622deaf51fd544240e6951e6b35ac16b942c6967f1588ef9b578be88f\": container with ID starting with 5a630af622deaf51fd544240e6951e6b35ac16b942c6967f1588ef9b578be88f not found: ID does not exist"
Oct 30 21:48:00.271644 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:48:00.271612    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-5jdb4"]
Oct 30 21:48:00.275147 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:48:00.275121    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-5jdb4"]
Oct 30 21:48:00.526958 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:48:00.526894    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32db9323-1a34-4407-9d58-9178bd288a9e" path="/var/lib/kubelet/pods/32db9323-1a34-4407-9d58-9178bd288a9e/volumes"
Oct 30 21:48:24.666036 ip-10-0-130-60 kubenswrapper[5061]: E1030 21:48:24.665992    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 21:48:54.665452 ip-10-0-130-60 kubenswrapper[5061]: E1030 21:48:54.665415    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 21:49:24.665830 ip-10-0-130-60 kubenswrapper[5061]: E1030 21:49:24.665796    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 21:49:31.720405 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:31.720370    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-94lvp"]
Oct 30 21:49:31.721041 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:31.721021    5061 cpu_manager.go:414] "containerMap: removing container" podUID="32db9323-1a34-4407-9d58-9178bd288a9e" containerName="authorino"
Oct 30 21:49:31.721084 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:31.721045    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32db9323-1a34-4407-9d58-9178bd288a9e" containerName="authorino"
Oct 30 21:49:31.721159 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:31.721147    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="32db9323-1a34-4407-9d58-9178bd288a9e" containerName="authorino"
Oct 30 21:49:31.724812 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:31.724790    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-94lvp"
Oct 30 21:49:31.791964 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:31.791932    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62dfda76-164c-4a47-9328-2468648266fd-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-94lvp\" (UID: \"62dfda76-164c-4a47-9328-2468648266fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-94lvp"
Oct 30 21:49:31.792101 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:31.791967    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62dfda76-164c-4a47-9328-2468648266fd-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-94lvp\" (UID: \"62dfda76-164c-4a47-9328-2468648266fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-94lvp"
Oct 30 21:49:31.792101 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:31.791989    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62dfda76-164c-4a47-9328-2468648266fd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-94lvp\" (UID: \"62dfda76-164c-4a47-9328-2468648266fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-94lvp"
Oct 30 21:49:31.792175 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:31.792119    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62dfda76-164c-4a47-9328-2468648266fd-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-94lvp\" (UID: \"62dfda76-164c-4a47-9328-2468648266fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-94lvp"
Oct 30 21:49:31.792175 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:31.792158    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62dfda76-164c-4a47-9328-2468648266fd-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-94lvp\" (UID: \"62dfda76-164c-4a47-9328-2468648266fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-94lvp"
Oct 30 21:49:31.792240 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:31.792212    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62dfda76-164c-4a47-9328-2468648266fd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-94lvp\" (UID: \"62dfda76-164c-4a47-9328-2468648266fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-94lvp"
Oct 30 21:49:31.792281 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:31.792267    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7xrd\" (UniqueName: \"kubernetes.io/projected/62dfda76-164c-4a47-9328-2468648266fd-kube-api-access-r7xrd\") pod \"maas-default-gateway-openshift-default-694db55c87-94lvp\" (UID: \"62dfda76-164c-4a47-9328-2468648266fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-94lvp"
Oct 30 21:49:31.792315 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:31.792292    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62dfda76-164c-4a47-9328-2468648266fd-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-94lvp\" (UID: \"62dfda76-164c-4a47-9328-2468648266fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-94lvp"
Oct 30 21:49:31.792354 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:31.792326    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62dfda76-164c-4a47-9328-2468648266fd-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-94lvp\" (UID: \"62dfda76-164c-4a47-9328-2468648266fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-94lvp"
Oct 30 21:49:31.893226 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:31.893196    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62dfda76-164c-4a47-9328-2468648266fd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-94lvp\" (UID: \"62dfda76-164c-4a47-9328-2468648266fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-94lvp"
Oct 30 21:49:31.893371 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:31.893236    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7xrd\" (UniqueName: \"kubernetes.io/projected/62dfda76-164c-4a47-9328-2468648266fd-kube-api-access-r7xrd\") pod \"maas-default-gateway-openshift-default-694db55c87-94lvp\" (UID: \"62dfda76-164c-4a47-9328-2468648266fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-94lvp"
Oct 30 21:49:31.893371 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:31.893261    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62dfda76-164c-4a47-9328-2468648266fd-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-94lvp\" (UID: \"62dfda76-164c-4a47-9328-2468648266fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-94lvp"
Oct 30 21:49:31.893371 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:31.893297    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62dfda76-164c-4a47-9328-2468648266fd-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-94lvp\" (UID: \"62dfda76-164c-4a47-9328-2468648266fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-94lvp"
Oct 30 21:49:31.893371 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:31.893323    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62dfda76-164c-4a47-9328-2468648266fd-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-94lvp\" (UID: \"62dfda76-164c-4a47-9328-2468648266fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-94lvp"
Oct 30 21:49:31.893371 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:31.893350    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62dfda76-164c-4a47-9328-2468648266fd-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-94lvp\" (UID: \"62dfda76-164c-4a47-9328-2468648266fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-94lvp"
Oct 30 21:49:31.893764 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:31.893437    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62dfda76-164c-4a47-9328-2468648266fd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-94lvp\" (UID: \"62dfda76-164c-4a47-9328-2468648266fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-94lvp"
Oct 30 21:49:31.893764 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:31.893509    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62dfda76-164c-4a47-9328-2468648266fd-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-94lvp\" (UID: \"62dfda76-164c-4a47-9328-2468648266fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-94lvp"
Oct 30 21:49:31.893764 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:31.893570    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62dfda76-164c-4a47-9328-2468648266fd-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-94lvp\" (UID: \"62dfda76-164c-4a47-9328-2468648266fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-94lvp"
Oct 30 21:49:31.893764 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:31.893728    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62dfda76-164c-4a47-9328-2468648266fd-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-94lvp\" (UID: \"62dfda76-164c-4a47-9328-2468648266fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-94lvp"
Oct 30 21:49:31.893764 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:31.893753    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62dfda76-164c-4a47-9328-2468648266fd-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-94lvp\" (UID: \"62dfda76-164c-4a47-9328-2468648266fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-94lvp"
Oct 30 21:49:31.894019 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:31.893887    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62dfda76-164c-4a47-9328-2468648266fd-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-94lvp\" (UID: \"62dfda76-164c-4a47-9328-2468648266fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-94lvp"
Oct 30 21:49:31.894019 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:31.893968    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62dfda76-164c-4a47-9328-2468648266fd-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-94lvp\" (UID: \"62dfda76-164c-4a47-9328-2468648266fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-94lvp"
Oct 30 21:49:31.894235 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:31.894216    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62dfda76-164c-4a47-9328-2468648266fd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-94lvp\" (UID: \"62dfda76-164c-4a47-9328-2468648266fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-94lvp"
Oct 30 21:49:31.895774 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:31.895746    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62dfda76-164c-4a47-9328-2468648266fd-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-94lvp\" (UID: \"62dfda76-164c-4a47-9328-2468648266fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-94lvp"
Oct 30 21:49:31.896011 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:31.895987    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62dfda76-164c-4a47-9328-2468648266fd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-94lvp\" (UID: \"62dfda76-164c-4a47-9328-2468648266fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-94lvp"
Oct 30 21:49:31.928949 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:31.928930    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-94lvp"]
Oct 30 21:49:31.951912 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:31.951879    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62dfda76-164c-4a47-9328-2468648266fd-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-94lvp\" (UID: \"62dfda76-164c-4a47-9328-2468648266fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-94lvp"
Oct 30 21:49:31.951912 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:31.951896    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7xrd\" (UniqueName: \"kubernetes.io/projected/62dfda76-164c-4a47-9328-2468648266fd-kube-api-access-r7xrd\") pod \"maas-default-gateway-openshift-default-694db55c87-94lvp\" (UID: \"62dfda76-164c-4a47-9328-2468648266fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-94lvp"
Oct 30 21:49:32.037924 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:32.037854    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-94lvp"
Oct 30 21:49:32.460461 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:32.460426    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-94lvp"]
Oct 30 21:49:32.465394 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:49:32.465368    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62dfda76_164c_4a47_9328_2468648266fd.slice/crio-71e182a04b195ab447238d66c602d8d9f4ca312a3880b8c50548031c0b06ce6e WatchSource:0}: Error finding container 71e182a04b195ab447238d66c602d8d9f4ca312a3880b8c50548031c0b06ce6e: Status 404 returned error can't find the container with id 71e182a04b195ab447238d66c602d8d9f4ca312a3880b8c50548031c0b06ce6e
Oct 30 21:49:32.467302 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:32.467269    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:49:32.467401 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:32.467350    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:49:32.467401 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:32.467394    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:49:32.529310 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:32.529272    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-94lvp" event={"ID":"62dfda76-164c-4a47-9328-2468648266fd","Type":"ContainerStarted","Data":"2016103dfa49cc28b3ea3fe995371d78fde903346c574a6487057308b6db36de"}
Oct 30 21:49:32.529437 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:32.529336    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-94lvp" event={"ID":"62dfda76-164c-4a47-9328-2468648266fd","Type":"ContainerStarted","Data":"71e182a04b195ab447238d66c602d8d9f4ca312a3880b8c50548031c0b06ce6e"}
Oct 30 21:49:32.643167 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:32.643102    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-94lvp" podStartSLOduration=1.643084364 podStartE2EDuration="1.643084364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:49:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:32.641836407 +0000 UTC m=+426.254606163" watchObservedRunningTime="2026-10-30 21:49:32.643084364 +0000 UTC m=+426.255854135"
Oct 30 21:49:33.045396 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:33.038018    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-94lvp"
Oct 30 21:49:33.045396 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:33.039709    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-94lvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body=
Oct 30 21:49:33.045396 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:33.039755    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-94lvp" podUID="62dfda76-164c-4a47-9328-2468648266fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused"
Oct 30 21:49:33.538206 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:33.538120    5061 generic.go:356] "Generic (PLEG): container finished" podID="99439df0-9c24-4a03-9dff-e95af6f3f3bd" containerID="40106f33775b1b36f0a7625562f0efa8af2bc9d98b0fe0f31067b2e8c776b69e" exitCode=0
Oct 30 21:49:33.538206 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:33.538194    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-58dhn" event={"ID":"99439df0-9c24-4a03-9dff-e95af6f3f3bd","Type":"ContainerDied","Data":"40106f33775b1b36f0a7625562f0efa8af2bc9d98b0fe0f31067b2e8c776b69e"}
Oct 30 21:49:33.538697 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:33.538677    5061 scope.go:122] "RemoveContainer" containerID="40106f33775b1b36f0a7625562f0efa8af2bc9d98b0fe0f31067b2e8c776b69e"
Oct 30 21:49:33.539753 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:33.539727    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:49:33.539828 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:33.539791    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:49:33.539828 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:33.539820    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:49:34.042550 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:34.042092    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-94lvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body=
Oct 30 21:49:34.042550 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:34.042164    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-94lvp" podUID="62dfda76-164c-4a47-9328-2468648266fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused"
Oct 30 21:49:34.565177 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:34.565142    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-58dhn" event={"ID":"99439df0-9c24-4a03-9dff-e95af6f3f3bd","Type":"ContainerStarted","Data":"a76a1f744b3dc65ae379f968708b9b3d2731dc2331f9249f74dd50d8cc36989c"}
Oct 30 21:49:35.039605 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:35.038954    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-94lvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body=
Oct 30 21:49:35.039605 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:35.039017    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-94lvp" podUID="62dfda76-164c-4a47-9328-2468648266fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused"
Oct 30 21:49:35.262901 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:35.262866    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-58dhn"
Oct 30 21:49:35.264215 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:35.264190    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-58dhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" start-of-body=
Oct 30 21:49:35.264340 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:35.264238    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-58dhn" podUID="99439df0-9c24-4a03-9dff-e95af6f3f3bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused"
Oct 30 21:49:36.041551 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:36.040945    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-94lvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body=
Oct 30 21:49:36.041551 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:36.041011    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-94lvp" podUID="62dfda76-164c-4a47-9328-2468648266fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused"
Oct 30 21:49:36.263647 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:36.263597    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-58dhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" start-of-body=
Oct 30 21:49:36.263812 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:36.263661    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-58dhn" podUID="99439df0-9c24-4a03-9dff-e95af6f3f3bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused"
Oct 30 21:49:37.039551 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:37.038759    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-94lvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body=
Oct 30 21:49:37.039551 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:37.038822    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-94lvp" podUID="62dfda76-164c-4a47-9328-2468648266fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused"
Oct 30 21:49:37.263264 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:37.263154    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-58dhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" start-of-body=
Oct 30 21:49:37.263264 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:37.263202    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-58dhn" podUID="99439df0-9c24-4a03-9dff-e95af6f3f3bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused"
Oct 30 21:49:37.579569 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:37.579488    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-58dhn"
Oct 30 21:49:38.041343 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:38.040859    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-94lvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body=
Oct 30 21:49:38.041343 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:38.040922    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-94lvp" podUID="62dfda76-164c-4a47-9328-2468648266fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused"
Oct 30 21:49:38.262853 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:38.262819    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-58dhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" start-of-body=
Oct 30 21:49:38.263015 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:38.262881    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-58dhn" podUID="99439df0-9c24-4a03-9dff-e95af6f3f3bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused"
Oct 30 21:49:39.042749 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:39.042717    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-94lvp"
Oct 30 21:49:39.043156 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:39.043034    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-94lvp"
Oct 30 21:49:39.043676 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:39.043658    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-94lvp"
Oct 30 21:49:39.263124 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:39.263091    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-58dhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" start-of-body=
Oct 30 21:49:39.263286 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:39.263141    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-58dhn" podUID="99439df0-9c24-4a03-9dff-e95af6f3f3bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused"
Oct 30 21:49:39.280035 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:39.280007    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-fcd8l"]
Oct 30 21:49:39.283401 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:39.283380    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-fcd8l"
Oct 30 21:49:39.347346 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:39.347311    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-fcd8l"]
Oct 30 21:49:39.402853 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:39.402815    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnjxp\" (UniqueName: \"kubernetes.io/projected/39d54039-6e50-4313-8172-1aaca8c67a93-kube-api-access-rnjxp\") pod \"authorino-6f6944bcdd-fcd8l\" (UID: \"39d54039-6e50-4313-8172-1aaca8c67a93\") " pod="kuadrant-system/authorino-6f6944bcdd-fcd8l"
Oct 30 21:49:39.503851 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:39.503806    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnjxp\" (UniqueName: \"kubernetes.io/projected/39d54039-6e50-4313-8172-1aaca8c67a93-kube-api-access-rnjxp\") pod \"authorino-6f6944bcdd-fcd8l\" (UID: \"39d54039-6e50-4313-8172-1aaca8c67a93\") " pod="kuadrant-system/authorino-6f6944bcdd-fcd8l"
Oct 30 21:49:39.561586 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:39.561522    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnjxp\" (UniqueName: \"kubernetes.io/projected/39d54039-6e50-4313-8172-1aaca8c67a93-kube-api-access-rnjxp\") pod \"authorino-6f6944bcdd-fcd8l\" (UID: \"39d54039-6e50-4313-8172-1aaca8c67a93\") " pod="kuadrant-system/authorino-6f6944bcdd-fcd8l"
Oct 30 21:49:39.594358 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:39.594331    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-fcd8l"
Oct 30 21:49:39.686867 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:39.686837    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-fcd8l"]
Oct 30 21:49:39.743099 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:39.743067    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6c57c78d95-mjhvv"]
Oct 30 21:49:39.746719 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:39.746701    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6c57c78d95-mjhvv"
Oct 30 21:49:39.806025 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:39.805999    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dchld\" (UniqueName: \"kubernetes.io/projected/7407db92-d055-4458-97d5-7429941c1777-kube-api-access-dchld\") pod \"authorino-6c57c78d95-mjhvv\" (UID: \"7407db92-d055-4458-97d5-7429941c1777\") " pod="kuadrant-system/authorino-6c57c78d95-mjhvv"
Oct 30 21:49:39.808579 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:39.808559    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6c57c78d95-mjhvv"]
Oct 30 21:49:39.907004 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:39.906930    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dchld\" (UniqueName: \"kubernetes.io/projected/7407db92-d055-4458-97d5-7429941c1777-kube-api-access-dchld\") pod \"authorino-6c57c78d95-mjhvv\" (UID: \"7407db92-d055-4458-97d5-7429941c1777\") " pod="kuadrant-system/authorino-6c57c78d95-mjhvv"
Oct 30 21:49:39.963808 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:39.963779    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dchld\" (UniqueName: \"kubernetes.io/projected/7407db92-d055-4458-97d5-7429941c1777-kube-api-access-dchld\") pod \"authorino-6c57c78d95-mjhvv\" (UID: \"7407db92-d055-4458-97d5-7429941c1777\") " pod="kuadrant-system/authorino-6c57c78d95-mjhvv"
Oct 30 21:49:39.965163 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:39.965144    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-fcd8l"]
Oct 30 21:49:39.971519 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:49:39.971476    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod39d54039_6e50_4313_8172_1aaca8c67a93.slice/crio-dc41980f71926fe3b5a9ded37d12fca30301345544023a4181109c7294380ade WatchSource:0}: Error finding container dc41980f71926fe3b5a9ded37d12fca30301345544023a4181109c7294380ade: Status 404 returned error can't find the container with id dc41980f71926fe3b5a9ded37d12fca30301345544023a4181109c7294380ade
Oct 30 21:49:40.058424 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:40.058398    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6c57c78d95-mjhvv"
Oct 30 21:49:40.067357 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:40.067336    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6c57c78d95-mjhvv"]
Oct 30 21:49:40.103154 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:40.103124    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-96689fcc7-ttkdx"]
Oct 30 21:49:40.107826 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:40.107803    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-96689fcc7-ttkdx"
Oct 30 21:49:40.159416 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:40.159359    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Oct 30 21:49:40.165582 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:40.165559    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-96689fcc7-ttkdx"]
Oct 30 21:49:40.209477 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:40.209454    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1831b1cd-70a7-4004-8fc7-0949179c0292-tls-cert\") pod \"authorino-96689fcc7-ttkdx\" (UID: \"1831b1cd-70a7-4004-8fc7-0949179c0292\") " pod="kuadrant-system/authorino-96689fcc7-ttkdx"
Oct 30 21:49:40.209610 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:40.209510    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsxs5\" (UniqueName: \"kubernetes.io/projected/1831b1cd-70a7-4004-8fc7-0949179c0292-kube-api-access-rsxs5\") pod \"authorino-96689fcc7-ttkdx\" (UID: \"1831b1cd-70a7-4004-8fc7-0949179c0292\") " pod="kuadrant-system/authorino-96689fcc7-ttkdx"
Oct 30 21:49:40.267044 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:40.267020    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-58dhn"
Oct 30 21:49:40.268001 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:40.267983    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-58dhn"
Oct 30 21:49:40.310659 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:40.310636    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1831b1cd-70a7-4004-8fc7-0949179c0292-tls-cert\") pod \"authorino-96689fcc7-ttkdx\" (UID: \"1831b1cd-70a7-4004-8fc7-0949179c0292\") " pod="kuadrant-system/authorino-96689fcc7-ttkdx"
Oct 30 21:49:40.310897 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:40.310829    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsxs5\" (UniqueName: \"kubernetes.io/projected/1831b1cd-70a7-4004-8fc7-0949179c0292-kube-api-access-rsxs5\") pod \"authorino-96689fcc7-ttkdx\" (UID: \"1831b1cd-70a7-4004-8fc7-0949179c0292\") " pod="kuadrant-system/authorino-96689fcc7-ttkdx"
Oct 30 21:49:40.313054 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:40.313037    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1831b1cd-70a7-4004-8fc7-0949179c0292-tls-cert\") pod \"authorino-96689fcc7-ttkdx\" (UID: \"1831b1cd-70a7-4004-8fc7-0949179c0292\") " pod="kuadrant-system/authorino-96689fcc7-ttkdx"
Oct 30 21:49:40.367695 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:40.367676    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsxs5\" (UniqueName: \"kubernetes.io/projected/1831b1cd-70a7-4004-8fc7-0949179c0292-kube-api-access-rsxs5\") pod \"authorino-96689fcc7-ttkdx\" (UID: \"1831b1cd-70a7-4004-8fc7-0949179c0292\") " pod="kuadrant-system/authorino-96689fcc7-ttkdx"
Oct 30 21:49:40.424891 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:40.424832    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-96689fcc7-ttkdx"
Oct 30 21:49:40.449025 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:40.449000    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6c57c78d95-mjhvv"]
Oct 30 21:49:40.454884 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:49:40.454860    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7407db92_d055_4458_97d5_7429941c1777.slice/crio-adb65e7157f10b34e2b6cfdccdd125443eb140f0e3e31ff8b0bf412b8a51e319 WatchSource:0}: Error finding container adb65e7157f10b34e2b6cfdccdd125443eb140f0e3e31ff8b0bf412b8a51e319: Status 404 returned error can't find the container with id adb65e7157f10b34e2b6cfdccdd125443eb140f0e3e31ff8b0bf412b8a51e319
Oct 30 21:49:40.600775 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:40.600746    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6c57c78d95-mjhvv" event={"ID":"7407db92-d055-4458-97d5-7429941c1777","Type":"ContainerStarted","Data":"adb65e7157f10b34e2b6cfdccdd125443eb140f0e3e31ff8b0bf412b8a51e319"}
Oct 30 21:49:40.602003 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:40.601972    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-fcd8l" event={"ID":"39d54039-6e50-4313-8172-1aaca8c67a93","Type":"ContainerStarted","Data":"dc41980f71926fe3b5a9ded37d12fca30301345544023a4181109c7294380ade"}
Oct 30 21:49:40.797448 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:40.797417    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-96689fcc7-ttkdx"]
Oct 30 21:49:40.803053 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:49:40.803018    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1831b1cd_70a7_4004_8fc7_0949179c0292.slice/crio-f83e12ecd1a83dfd1c2ce7d63785fc1fb66fa7f4af4743084d98b66a47631c88 WatchSource:0}: Error finding container f83e12ecd1a83dfd1c2ce7d63785fc1fb66fa7f4af4743084d98b66a47631c88: Status 404 returned error can't find the container with id f83e12ecd1a83dfd1c2ce7d63785fc1fb66fa7f4af4743084d98b66a47631c88
Oct 30 21:49:41.607980 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:41.607938    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-96689fcc7-ttkdx" event={"ID":"1831b1cd-70a7-4004-8fc7-0949179c0292","Type":"ContainerStarted","Data":"f83e12ecd1a83dfd1c2ce7d63785fc1fb66fa7f4af4743084d98b66a47631c88"}
Oct 30 21:49:42.628099 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:42.628063    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-96689fcc7-ttkdx" event={"ID":"1831b1cd-70a7-4004-8fc7-0949179c0292","Type":"ContainerStarted","Data":"3505d85a612ca508bd381506d93ee2af1d9340c45e85c96cb71bf29a7543d266"}
Oct 30 21:49:42.630815 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:42.630782    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6c57c78d95-mjhvv" podUID="7407db92-d055-4458-97d5-7429941c1777" containerName="authorino" containerID="cri-o://c7f1b08bbc778a691b8ae4fd2f6b82422c89cbf5d6fc0a4b6121d4bd9742056c" gracePeriod=30
Oct 30 21:49:42.631022 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:42.631002    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6c57c78d95-mjhvv" event={"ID":"7407db92-d055-4458-97d5-7429941c1777","Type":"ContainerStarted","Data":"c7f1b08bbc778a691b8ae4fd2f6b82422c89cbf5d6fc0a4b6121d4bd9742056c"}
Oct 30 21:49:42.633222 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:42.633193    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-fcd8l" event={"ID":"39d54039-6e50-4313-8172-1aaca8c67a93","Type":"ContainerStarted","Data":"689f7a74021287b9379c677e96222d451e1db8a6126b704dc853f75670c2a64e"}
Oct 30 21:49:42.633337 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:42.633261    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6f6944bcdd-fcd8l" podUID="39d54039-6e50-4313-8172-1aaca8c67a93" containerName="authorino" containerID="cri-o://689f7a74021287b9379c677e96222d451e1db8a6126b704dc853f75670c2a64e" gracePeriod=30
Oct 30 21:49:42.740230 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:42.740177    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-96689fcc7-ttkdx" podStartSLOduration=1.40299574 podStartE2EDuration="2.740163971s" totalImagesPullingTime="1.337168231s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:49:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:42.739229051 +0000 UTC m=+436.351998810" watchObservedRunningTime="2026-10-30 21:49:42.740163971 +0000 UTC m=+436.352933727"
Oct 30 21:49:42.779411 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:42.779355    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-px4hr"]
Oct 30 21:49:42.782574 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:42.780274    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-px4hr" podUID="3011467a-326a-4acb-9501-9175ed4cab27" containerName="authorino" containerID="cri-o://3a1d377717328f4fcb8784632cdee0673818f9aae3e366c4c3a874ad1eedf8fd" gracePeriod=30
Oct 30 21:49:42.859250 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:42.859192    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6f6944bcdd-fcd8l" podStartSLOduration=2.191200294 podStartE2EDuration="3.859172223s" totalImagesPullingTime="1.667971929s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:49:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:42.857644342 +0000 UTC m=+436.470414085" watchObservedRunningTime="2026-10-30 21:49:42.859172223 +0000 UTC m=+436.471941981"
Oct 30 21:49:42.947213 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:42.947189    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-fcd8l"
Oct 30 21:49:42.950320 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:42.950254    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6c57c78d95-mjhvv"
Oct 30 21:49:42.973880 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:42.973837    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6c57c78d95-mjhvv" podStartSLOduration=2.568965314 podStartE2EDuration="3.973824977s" totalImagesPullingTime="1.404859663s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:49:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:42.972483143 +0000 UTC m=+436.585252899" watchObservedRunningTime="2026-10-30 21:49:42.973824977 +0000 UTC m=+436.586594733"
Oct 30 21:49:43.036333 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:43.036298    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39d54039-6e50-4313-8172-1aaca8c67a93-kube-api-access-rnjxp\" (UniqueName: \"kubernetes.io/projected/39d54039-6e50-4313-8172-1aaca8c67a93-kube-api-access-rnjxp\") pod \"39d54039-6e50-4313-8172-1aaca8c67a93\" (UID: \"39d54039-6e50-4313-8172-1aaca8c67a93\") "
Oct 30 21:49:43.036555 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:43.036447    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7407db92-d055-4458-97d5-7429941c1777-kube-api-access-dchld\" (UniqueName: \"kubernetes.io/projected/7407db92-d055-4458-97d5-7429941c1777-kube-api-access-dchld\") pod \"7407db92-d055-4458-97d5-7429941c1777\" (UID: \"7407db92-d055-4458-97d5-7429941c1777\") "
Oct 30 21:49:43.038437 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:43.038401    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39d54039-6e50-4313-8172-1aaca8c67a93-kube-api-access-rnjxp" pod "39d54039-6e50-4313-8172-1aaca8c67a93" (UID: "39d54039-6e50-4313-8172-1aaca8c67a93"). InnerVolumeSpecName "kube-api-access-rnjxp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:49:43.038570 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:43.038509    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7407db92-d055-4458-97d5-7429941c1777-kube-api-access-dchld" pod "7407db92-d055-4458-97d5-7429941c1777" (UID: "7407db92-d055-4458-97d5-7429941c1777"). InnerVolumeSpecName "kube-api-access-dchld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:49:43.058481 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:43.058459    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-px4hr"
Oct 30 21:49:43.137957 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:43.137924    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3011467a-326a-4acb-9501-9175ed4cab27-kube-api-access-h85sl\" (UniqueName: \"kubernetes.io/projected/3011467a-326a-4acb-9501-9175ed4cab27-kube-api-access-h85sl\") pod \"3011467a-326a-4acb-9501-9175ed4cab27\" (UID: \"3011467a-326a-4acb-9501-9175ed4cab27\") "
Oct 30 21:49:43.138156 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:43.138144    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnjxp\" (UniqueName: \"kubernetes.io/projected/39d54039-6e50-4313-8172-1aaca8c67a93-kube-api-access-rnjxp\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:49:43.138197 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:43.138158    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dchld\" (UniqueName: \"kubernetes.io/projected/7407db92-d055-4458-97d5-7429941c1777-kube-api-access-dchld\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:49:43.140189 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:43.140153    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3011467a-326a-4acb-9501-9175ed4cab27-kube-api-access-h85sl" pod "3011467a-326a-4acb-9501-9175ed4cab27" (UID: "3011467a-326a-4acb-9501-9175ed4cab27"). InnerVolumeSpecName "kube-api-access-h85sl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:49:43.239361 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:43.239288    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h85sl\" (UniqueName: \"kubernetes.io/projected/3011467a-326a-4acb-9501-9175ed4cab27-kube-api-access-h85sl\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:49:43.639805 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:43.639768    5061 generic.go:356] "Generic (PLEG): container finished" podID="3011467a-326a-4acb-9501-9175ed4cab27" containerID="3a1d377717328f4fcb8784632cdee0673818f9aae3e366c4c3a874ad1eedf8fd" exitCode=0
Oct 30 21:49:43.640292 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:43.639818    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-px4hr"
Oct 30 21:49:43.640292 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:43.639844    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-px4hr" event={"ID":"3011467a-326a-4acb-9501-9175ed4cab27","Type":"ContainerDied","Data":"3a1d377717328f4fcb8784632cdee0673818f9aae3e366c4c3a874ad1eedf8fd"}
Oct 30 21:49:43.640292 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:43.639880    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-px4hr" event={"ID":"3011467a-326a-4acb-9501-9175ed4cab27","Type":"ContainerDied","Data":"9120ef374bf38958b2985109b6a87794258416f05ca7450ad96c63e1b773f08d"}
Oct 30 21:49:43.640292 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:43.639904    5061 scope.go:122] "RemoveContainer" containerID="3a1d377717328f4fcb8784632cdee0673818f9aae3e366c4c3a874ad1eedf8fd"
Oct 30 21:49:43.641202 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:43.641182    5061 generic.go:356] "Generic (PLEG): container finished" podID="39d54039-6e50-4313-8172-1aaca8c67a93" containerID="689f7a74021287b9379c677e96222d451e1db8a6126b704dc853f75670c2a64e" exitCode=0
Oct 30 21:49:43.641322 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:43.641222    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-fcd8l"
Oct 30 21:49:43.641322 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:43.641265    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-fcd8l" event={"ID":"39d54039-6e50-4313-8172-1aaca8c67a93","Type":"ContainerDied","Data":"689f7a74021287b9379c677e96222d451e1db8a6126b704dc853f75670c2a64e"}
Oct 30 21:49:43.641322 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:43.641309    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-fcd8l" event={"ID":"39d54039-6e50-4313-8172-1aaca8c67a93","Type":"ContainerDied","Data":"dc41980f71926fe3b5a9ded37d12fca30301345544023a4181109c7294380ade"}
Oct 30 21:49:43.642869 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:43.642776    5061 generic.go:356] "Generic (PLEG): container finished" podID="7407db92-d055-4458-97d5-7429941c1777" containerID="c7f1b08bbc778a691b8ae4fd2f6b82422c89cbf5d6fc0a4b6121d4bd9742056c" exitCode=0
Oct 30 21:49:43.642869 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:43.642800    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6c57c78d95-mjhvv" event={"ID":"7407db92-d055-4458-97d5-7429941c1777","Type":"ContainerDied","Data":"c7f1b08bbc778a691b8ae4fd2f6b82422c89cbf5d6fc0a4b6121d4bd9742056c"}
Oct 30 21:49:43.642869 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:43.642823    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6c57c78d95-mjhvv"
Oct 30 21:49:43.642869 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:43.642842    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6c57c78d95-mjhvv" event={"ID":"7407db92-d055-4458-97d5-7429941c1777","Type":"ContainerDied","Data":"adb65e7157f10b34e2b6cfdccdd125443eb140f0e3e31ff8b0bf412b8a51e319"}
Oct 30 21:49:43.649420 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:43.649402    5061 scope.go:122] "RemoveContainer" containerID="3a1d377717328f4fcb8784632cdee0673818f9aae3e366c4c3a874ad1eedf8fd"
Oct 30 21:49:43.649718 ip-10-0-130-60 kubenswrapper[5061]: E1030 21:49:43.649689    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a1d377717328f4fcb8784632cdee0673818f9aae3e366c4c3a874ad1eedf8fd\": container with ID starting with 3a1d377717328f4fcb8784632cdee0673818f9aae3e366c4c3a874ad1eedf8fd not found: ID does not exist" containerID="3a1d377717328f4fcb8784632cdee0673818f9aae3e366c4c3a874ad1eedf8fd"
Oct 30 21:49:43.649826 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:43.649724    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a1d377717328f4fcb8784632cdee0673818f9aae3e366c4c3a874ad1eedf8fd"} err="failed to get container status \"3a1d377717328f4fcb8784632cdee0673818f9aae3e366c4c3a874ad1eedf8fd\": rpc error: code = NotFound desc = could not find container \"3a1d377717328f4fcb8784632cdee0673818f9aae3e366c4c3a874ad1eedf8fd\": container with ID starting with 3a1d377717328f4fcb8784632cdee0673818f9aae3e366c4c3a874ad1eedf8fd not found: ID does not exist"
Oct 30 21:49:43.649826 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:43.649748    5061 scope.go:122] "RemoveContainer" containerID="689f7a74021287b9379c677e96222d451e1db8a6126b704dc853f75670c2a64e"
Oct 30 21:49:43.658238 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:43.658222    5061 scope.go:122] "RemoveContainer" containerID="689f7a74021287b9379c677e96222d451e1db8a6126b704dc853f75670c2a64e"
Oct 30 21:49:43.658487 ip-10-0-130-60 kubenswrapper[5061]: E1030 21:49:43.658471    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"689f7a74021287b9379c677e96222d451e1db8a6126b704dc853f75670c2a64e\": container with ID starting with 689f7a74021287b9379c677e96222d451e1db8a6126b704dc853f75670c2a64e not found: ID does not exist" containerID="689f7a74021287b9379c677e96222d451e1db8a6126b704dc853f75670c2a64e"
Oct 30 21:49:43.658572 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:43.658491    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"689f7a74021287b9379c677e96222d451e1db8a6126b704dc853f75670c2a64e"} err="failed to get container status \"689f7a74021287b9379c677e96222d451e1db8a6126b704dc853f75670c2a64e\": rpc error: code = NotFound desc = could not find container \"689f7a74021287b9379c677e96222d451e1db8a6126b704dc853f75670c2a64e\": container with ID starting with 689f7a74021287b9379c677e96222d451e1db8a6126b704dc853f75670c2a64e not found: ID does not exist"
Oct 30 21:49:43.658572 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:43.658505    5061 scope.go:122] "RemoveContainer" containerID="c7f1b08bbc778a691b8ae4fd2f6b82422c89cbf5d6fc0a4b6121d4bd9742056c"
Oct 30 21:49:43.665748 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:43.665721    5061 scope.go:122] "RemoveContainer" containerID="c7f1b08bbc778a691b8ae4fd2f6b82422c89cbf5d6fc0a4b6121d4bd9742056c"
Oct 30 21:49:43.666012 ip-10-0-130-60 kubenswrapper[5061]: E1030 21:49:43.665993    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7f1b08bbc778a691b8ae4fd2f6b82422c89cbf5d6fc0a4b6121d4bd9742056c\": container with ID starting with c7f1b08bbc778a691b8ae4fd2f6b82422c89cbf5d6fc0a4b6121d4bd9742056c not found: ID does not exist" containerID="c7f1b08bbc778a691b8ae4fd2f6b82422c89cbf5d6fc0a4b6121d4bd9742056c"
Oct 30 21:49:43.666066 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:43.666016    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7f1b08bbc778a691b8ae4fd2f6b82422c89cbf5d6fc0a4b6121d4bd9742056c"} err="failed to get container status \"c7f1b08bbc778a691b8ae4fd2f6b82422c89cbf5d6fc0a4b6121d4bd9742056c\": rpc error: code = NotFound desc = could not find container \"c7f1b08bbc778a691b8ae4fd2f6b82422c89cbf5d6fc0a4b6121d4bd9742056c\": container with ID starting with c7f1b08bbc778a691b8ae4fd2f6b82422c89cbf5d6fc0a4b6121d4bd9742056c not found: ID does not exist"
Oct 30 21:49:43.813385 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:43.813358    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-px4hr"]
Oct 30 21:49:43.828624 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:43.828598    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-px4hr"]
Oct 30 21:49:43.934790 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:43.934725    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-fcd8l"]
Oct 30 21:49:43.943819 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:43.943797    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-fcd8l"]
Oct 30 21:49:44.052791 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:44.052766    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6c57c78d95-mjhvv"]
Oct 30 21:49:44.070857 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:44.070833    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6c57c78d95-mjhvv"]
Oct 30 21:49:44.526228 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:44.526199    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3011467a-326a-4acb-9501-9175ed4cab27" path="/var/lib/kubelet/pods/3011467a-326a-4acb-9501-9175ed4cab27/volumes"
Oct 30 21:49:44.526566 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:44.526551    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39d54039-6e50-4313-8172-1aaca8c67a93" path="/var/lib/kubelet/pods/39d54039-6e50-4313-8172-1aaca8c67a93/volumes"
Oct 30 21:49:44.526845 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:49:44.526834    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7407db92-d055-4458-97d5-7429941c1777" path="/var/lib/kubelet/pods/7407db92-d055-4458-97d5-7429941c1777/volumes"
Oct 30 21:49:54.664786 ip-10-0-130-60 kubenswrapper[5061]: E1030 21:49:54.664745    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 21:50:24.664230 ip-10-0-130-60 kubenswrapper[5061]: E1030 21:50:24.664196    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 21:50:54.664977 ip-10-0-130-60 kubenswrapper[5061]: E1030 21:50:54.664901    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 21:51:01.770833 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:01.770798    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-ggrcq"]
Oct 30 21:51:01.771331 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:01.771184    5061 cpu_manager.go:414] "containerMap: removing container" podUID="7407db92-d055-4458-97d5-7429941c1777" containerName="authorino"
Oct 30 21:51:01.771331 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:01.771197    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7407db92-d055-4458-97d5-7429941c1777" containerName="authorino"
Oct 30 21:51:01.771331 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:01.771208    5061 cpu_manager.go:414] "containerMap: removing container" podUID="3011467a-326a-4acb-9501-9175ed4cab27" containerName="authorino"
Oct 30 21:51:01.771331 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:01.771213    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3011467a-326a-4acb-9501-9175ed4cab27" containerName="authorino"
Oct 30 21:51:01.771331 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:01.771243    5061 cpu_manager.go:414] "containerMap: removing container" podUID="39d54039-6e50-4313-8172-1aaca8c67a93" containerName="authorino"
Oct 30 21:51:01.771331 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:01.771248    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39d54039-6e50-4313-8172-1aaca8c67a93" containerName="authorino"
Oct 30 21:51:01.771331 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:01.771323    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="39d54039-6e50-4313-8172-1aaca8c67a93" containerName="authorino"
Oct 30 21:51:01.771331 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:01.771331    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="7407db92-d055-4458-97d5-7429941c1777" containerName="authorino"
Oct 30 21:51:01.771331 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:01.771338    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="3011467a-326a-4acb-9501-9175ed4cab27" containerName="authorino"
Oct 30 21:51:01.774727 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:01.774705    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-ggrcq"
Oct 30 21:51:01.824760 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:01.824732    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Oct 30 21:51:01.824905 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:01.824781    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Oct 30 21:51:01.871997 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:01.871971    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-sm8jq\""
Oct 30 21:51:01.872246 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:01.872227    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Oct 30 21:51:01.873887 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:01.873869    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-ggrcq"]
Oct 30 21:51:01.903136 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:01.903103    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkdlw\" (UniqueName: \"kubernetes.io/projected/cd4aa9fd-71de-4b7c-8c45-d7e41cf0ba9b-kube-api-access-rkdlw\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-ggrcq\" (UID: \"cd4aa9fd-71de-4b7c-8c45-d7e41cf0ba9b\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-ggrcq"
Oct 30 21:51:01.903271 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:01.903145    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd4aa9fd-71de-4b7c-8c45-d7e41cf0ba9b-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-ggrcq\" (UID: \"cd4aa9fd-71de-4b7c-8c45-d7e41cf0ba9b\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-ggrcq"
Oct 30 21:51:01.903271 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:01.903216    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd4aa9fd-71de-4b7c-8c45-d7e41cf0ba9b-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-ggrcq\" (UID: \"cd4aa9fd-71de-4b7c-8c45-d7e41cf0ba9b\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-ggrcq"
Oct 30 21:51:01.903342 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:01.903282    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd4aa9fd-71de-4b7c-8c45-d7e41cf0ba9b-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-ggrcq\" (UID: \"cd4aa9fd-71de-4b7c-8c45-d7e41cf0ba9b\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-ggrcq"
Oct 30 21:51:01.903342 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:01.903300    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd4aa9fd-71de-4b7c-8c45-d7e41cf0ba9b-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-ggrcq\" (UID: \"cd4aa9fd-71de-4b7c-8c45-d7e41cf0ba9b\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-ggrcq"
Oct 30 21:51:01.903407 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:01.903363    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd4aa9fd-71de-4b7c-8c45-d7e41cf0ba9b-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-ggrcq\" (UID: \"cd4aa9fd-71de-4b7c-8c45-d7e41cf0ba9b\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-ggrcq"
Oct 30 21:51:02.006958 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:02.006919    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkdlw\" (UniqueName: \"kubernetes.io/projected/cd4aa9fd-71de-4b7c-8c45-d7e41cf0ba9b-kube-api-access-rkdlw\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-ggrcq\" (UID: \"cd4aa9fd-71de-4b7c-8c45-d7e41cf0ba9b\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-ggrcq"
Oct 30 21:51:02.007128 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:02.006976    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd4aa9fd-71de-4b7c-8c45-d7e41cf0ba9b-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-ggrcq\" (UID: \"cd4aa9fd-71de-4b7c-8c45-d7e41cf0ba9b\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-ggrcq"
Oct 30 21:51:02.007177 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:02.007127    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd4aa9fd-71de-4b7c-8c45-d7e41cf0ba9b-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-ggrcq\" (UID: \"cd4aa9fd-71de-4b7c-8c45-d7e41cf0ba9b\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-ggrcq"
Oct 30 21:51:02.007177 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:02.007164    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd4aa9fd-71de-4b7c-8c45-d7e41cf0ba9b-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-ggrcq\" (UID: \"cd4aa9fd-71de-4b7c-8c45-d7e41cf0ba9b\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-ggrcq"
Oct 30 21:51:02.007289 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:02.007181    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd4aa9fd-71de-4b7c-8c45-d7e41cf0ba9b-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-ggrcq\" (UID: \"cd4aa9fd-71de-4b7c-8c45-d7e41cf0ba9b\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-ggrcq"
Oct 30 21:51:02.007289 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:02.007201    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd4aa9fd-71de-4b7c-8c45-d7e41cf0ba9b-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-ggrcq\" (UID: \"cd4aa9fd-71de-4b7c-8c45-d7e41cf0ba9b\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-ggrcq"
Oct 30 21:51:02.007593 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:02.007568    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd4aa9fd-71de-4b7c-8c45-d7e41cf0ba9b-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-ggrcq\" (UID: \"cd4aa9fd-71de-4b7c-8c45-d7e41cf0ba9b\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-ggrcq"
Oct 30 21:51:02.007776 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:02.007596    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd4aa9fd-71de-4b7c-8c45-d7e41cf0ba9b-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-ggrcq\" (UID: \"cd4aa9fd-71de-4b7c-8c45-d7e41cf0ba9b\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-ggrcq"
Oct 30 21:51:02.007776 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:02.007634    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd4aa9fd-71de-4b7c-8c45-d7e41cf0ba9b-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-ggrcq\" (UID: \"cd4aa9fd-71de-4b7c-8c45-d7e41cf0ba9b\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-ggrcq"
Oct 30 21:51:02.009477 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:02.009457    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd4aa9fd-71de-4b7c-8c45-d7e41cf0ba9b-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-ggrcq\" (UID: \"cd4aa9fd-71de-4b7c-8c45-d7e41cf0ba9b\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-ggrcq"
Oct 30 21:51:02.009589 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:02.009549    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd4aa9fd-71de-4b7c-8c45-d7e41cf0ba9b-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-ggrcq\" (UID: \"cd4aa9fd-71de-4b7c-8c45-d7e41cf0ba9b\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-ggrcq"
Oct 30 21:51:02.063122 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:02.063093    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkdlw\" (UniqueName: \"kubernetes.io/projected/cd4aa9fd-71de-4b7c-8c45-d7e41cf0ba9b-kube-api-access-rkdlw\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-ggrcq\" (UID: \"cd4aa9fd-71de-4b7c-8c45-d7e41cf0ba9b\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-ggrcq"
Oct 30 21:51:02.087565 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:02.087521    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-ggrcq"
Oct 30 21:51:02.453517 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:02.453446    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-ggrcq"]
Oct 30 21:51:02.458869 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:51:02.458839    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd4aa9fd_71de_4b7c_8c45_d7e41cf0ba9b.slice/crio-b21610e1f446ede18e48ca0072674ee3359e45903dc529db1bd8d8ad3e05a552 WatchSource:0}: Error finding container b21610e1f446ede18e48ca0072674ee3359e45903dc529db1bd8d8ad3e05a552: Status 404 returned error can't find the container with id b21610e1f446ede18e48ca0072674ee3359e45903dc529db1bd8d8ad3e05a552
Oct 30 21:51:03.007737 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:03.007697    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-ggrcq" event={"ID":"cd4aa9fd-71de-4b7c-8c45-d7e41cf0ba9b","Type":"ContainerStarted","Data":"b21610e1f446ede18e48ca0072674ee3359e45903dc529db1bd8d8ad3e05a552"}
Oct 30 21:51:05.025791 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:05.025738    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-ggrcq" event={"ID":"cd4aa9fd-71de-4b7c-8c45-d7e41cf0ba9b","Type":"ContainerStarted","Data":"b526f11522b7d13a2353b13491510a654a24d7c3df575155aaf43052373dd65a"}
Oct 30 21:51:05.088641 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:05.088598    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-ggrcq"
Oct 30 21:51:05.138117 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:05.138055    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-ggrcq" podStartSLOduration=2.593129755 podStartE2EDuration="4.138040997s" totalImagesPullingTime="1.544911242s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:51:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:51:05.136631481 +0000 UTC m=+518.749401237" watchObservedRunningTime="2026-10-30 21:51:05.138040997 +0000 UTC m=+518.750810750"
Oct 30 21:51:09.890572 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:09.890514    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-hfpzd"]
Oct 30 21:51:09.895244 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:09.895214    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-hfpzd"
Oct 30 21:51:09.981010 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:09.980969    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f552c94-29fe-485b-9410-5419467dd6ac-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-hfpzd\" (UID: \"8f552c94-29fe-485b-9410-5419467dd6ac\") " pod="llm/echo-simulator-kserve-64b6f8f96c-hfpzd"
Oct 30 21:51:09.981010 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:09.981012    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f552c94-29fe-485b-9410-5419467dd6ac-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-hfpzd\" (UID: \"8f552c94-29fe-485b-9410-5419467dd6ac\") " pod="llm/echo-simulator-kserve-64b6f8f96c-hfpzd"
Oct 30 21:51:09.981220 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:09.981043    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f552c94-29fe-485b-9410-5419467dd6ac-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-hfpzd\" (UID: \"8f552c94-29fe-485b-9410-5419467dd6ac\") " pod="llm/echo-simulator-kserve-64b6f8f96c-hfpzd"
Oct 30 21:51:09.981292 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:09.981264    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f552c94-29fe-485b-9410-5419467dd6ac-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-hfpzd\" (UID: \"8f552c94-29fe-485b-9410-5419467dd6ac\") " pod="llm/echo-simulator-kserve-64b6f8f96c-hfpzd"
Oct 30 21:51:09.981417 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:09.981399    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvbsp\" (UniqueName: \"kubernetes.io/projected/8f552c94-29fe-485b-9410-5419467dd6ac-kube-api-access-kvbsp\") pod \"echo-simulator-kserve-64b6f8f96c-hfpzd\" (UID: \"8f552c94-29fe-485b-9410-5419467dd6ac\") " pod="llm/echo-simulator-kserve-64b6f8f96c-hfpzd"
Oct 30 21:51:09.981482 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:09.981431    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f552c94-29fe-485b-9410-5419467dd6ac-home\") pod \"echo-simulator-kserve-64b6f8f96c-hfpzd\" (UID: \"8f552c94-29fe-485b-9410-5419467dd6ac\") " pod="llm/echo-simulator-kserve-64b6f8f96c-hfpzd"
Oct 30 21:51:09.992374 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:09.992348    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\""
Oct 30 21:51:09.994207 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:09.994183    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-hfpzd"]
Oct 30 21:51:10.082921 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:10.082888    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f552c94-29fe-485b-9410-5419467dd6ac-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-hfpzd\" (UID: \"8f552c94-29fe-485b-9410-5419467dd6ac\") " pod="llm/echo-simulator-kserve-64b6f8f96c-hfpzd"
Oct 30 21:51:10.083099 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:10.083074    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvbsp\" (UniqueName: \"kubernetes.io/projected/8f552c94-29fe-485b-9410-5419467dd6ac-kube-api-access-kvbsp\") pod \"echo-simulator-kserve-64b6f8f96c-hfpzd\" (UID: \"8f552c94-29fe-485b-9410-5419467dd6ac\") " pod="llm/echo-simulator-kserve-64b6f8f96c-hfpzd"
Oct 30 21:51:10.083155 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:10.083120    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f552c94-29fe-485b-9410-5419467dd6ac-home\") pod \"echo-simulator-kserve-64b6f8f96c-hfpzd\" (UID: \"8f552c94-29fe-485b-9410-5419467dd6ac\") " pod="llm/echo-simulator-kserve-64b6f8f96c-hfpzd"
Oct 30 21:51:10.083211 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:10.083157    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f552c94-29fe-485b-9410-5419467dd6ac-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-hfpzd\" (UID: \"8f552c94-29fe-485b-9410-5419467dd6ac\") " pod="llm/echo-simulator-kserve-64b6f8f96c-hfpzd"
Oct 30 21:51:10.083211 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:10.083182    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f552c94-29fe-485b-9410-5419467dd6ac-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-hfpzd\" (UID: \"8f552c94-29fe-485b-9410-5419467dd6ac\") " pod="llm/echo-simulator-kserve-64b6f8f96c-hfpzd"
Oct 30 21:51:10.083315 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:10.083220    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f552c94-29fe-485b-9410-5419467dd6ac-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-hfpzd\" (UID: \"8f552c94-29fe-485b-9410-5419467dd6ac\") " pod="llm/echo-simulator-kserve-64b6f8f96c-hfpzd"
Oct 30 21:51:10.083420 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:10.083397    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f552c94-29fe-485b-9410-5419467dd6ac-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-hfpzd\" (UID: \"8f552c94-29fe-485b-9410-5419467dd6ac\") " pod="llm/echo-simulator-kserve-64b6f8f96c-hfpzd"
Oct 30 21:51:10.083687 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:10.083667    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f552c94-29fe-485b-9410-5419467dd6ac-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-hfpzd\" (UID: \"8f552c94-29fe-485b-9410-5419467dd6ac\") " pod="llm/echo-simulator-kserve-64b6f8f96c-hfpzd"
Oct 30 21:51:10.084070 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:10.084046    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f552c94-29fe-485b-9410-5419467dd6ac-home\") pod \"echo-simulator-kserve-64b6f8f96c-hfpzd\" (UID: \"8f552c94-29fe-485b-9410-5419467dd6ac\") " pod="llm/echo-simulator-kserve-64b6f8f96c-hfpzd"
Oct 30 21:51:10.085988 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:10.085964    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f552c94-29fe-485b-9410-5419467dd6ac-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-hfpzd\" (UID: \"8f552c94-29fe-485b-9410-5419467dd6ac\") " pod="llm/echo-simulator-kserve-64b6f8f96c-hfpzd"
Oct 30 21:51:10.086326 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:10.086284    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f552c94-29fe-485b-9410-5419467dd6ac-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-hfpzd\" (UID: \"8f552c94-29fe-485b-9410-5419467dd6ac\") " pod="llm/echo-simulator-kserve-64b6f8f96c-hfpzd"
Oct 30 21:51:10.138625 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:10.138591    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvbsp\" (UniqueName: \"kubernetes.io/projected/8f552c94-29fe-485b-9410-5419467dd6ac-kube-api-access-kvbsp\") pod \"echo-simulator-kserve-64b6f8f96c-hfpzd\" (UID: \"8f552c94-29fe-485b-9410-5419467dd6ac\") " pod="llm/echo-simulator-kserve-64b6f8f96c-hfpzd"
Oct 30 21:51:10.210233 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:10.210143    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-hfpzd"
Oct 30 21:51:10.675550 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:10.674983    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-hfpzd"]
Oct 30 21:51:10.677409 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:51:10.677373    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f552c94_29fe_485b_9410_5419467dd6ac.slice/crio-81ad063d3475f672e24676ec3171a310ff849a97dcc1bc314e07c9eebc9d478b WatchSource:0}: Error finding container 81ad063d3475f672e24676ec3171a310ff849a97dcc1bc314e07c9eebc9d478b: Status 404 returned error can't find the container with id 81ad063d3475f672e24676ec3171a310ff849a97dcc1bc314e07c9eebc9d478b
Oct 30 21:51:11.058570 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:11.058514    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-hfpzd" event={"ID":"8f552c94-29fe-485b-9410-5419467dd6ac","Type":"ContainerStarted","Data":"81ad063d3475f672e24676ec3171a310ff849a97dcc1bc314e07c9eebc9d478b"}
Oct 30 21:51:12.088131 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:12.088102    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-ggrcq"
Oct 30 21:51:12.103186 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:12.103151    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-ggrcq"
Oct 30 21:51:13.082463 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:13.082437    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-ggrcq"
Oct 30 21:51:14.077514 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:14.077473    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-hfpzd" event={"ID":"8f552c94-29fe-485b-9410-5419467dd6ac","Type":"ContainerStarted","Data":"63b0993089c12a86f84cc0dc94e4baf856b1418ef71fc106347c21d63ab51c41"}
Oct 30 21:51:14.200207 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:14.200140    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-hfpzd" podStartSLOduration=2.746896673 podStartE2EDuration="5.200126481s" totalImagesPullingTime="2.453229808s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:51:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:51:14.197906067 +0000 UTC m=+527.810675823" watchObservedRunningTime="2026-10-30 21:51:14.200126481 +0000 UTC m=+527.812896238"
Oct 30 21:51:14.210649 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:14.210612    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-hfpzd"
Oct 30 21:51:15.932797 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:15.932764    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-blrrb"]
Oct 30 21:51:15.936493 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:15.936473    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-blrrb"
Oct 30 21:51:16.005736 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:16.005713    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Oct 30 21:51:16.035164 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:16.035135    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-blrrb"]
Oct 30 21:51:16.044922 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:16.044903    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb4b1ed5-2192-4877-8be7-ca82af831af5-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-blrrb\" (UID: \"cb4b1ed5-2192-4877-8be7-ca82af831af5\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-blrrb"
Oct 30 21:51:16.045006 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:16.044937    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8ckt\" (UniqueName: \"kubernetes.io/projected/cb4b1ed5-2192-4877-8be7-ca82af831af5-kube-api-access-r8ckt\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-blrrb\" (UID: \"cb4b1ed5-2192-4877-8be7-ca82af831af5\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-blrrb"
Oct 30 21:51:16.045051 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:16.045030    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb4b1ed5-2192-4877-8be7-ca82af831af5-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-blrrb\" (UID: \"cb4b1ed5-2192-4877-8be7-ca82af831af5\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-blrrb"
Oct 30 21:51:16.045113 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:16.045100    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb4b1ed5-2192-4877-8be7-ca82af831af5-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-blrrb\" (UID: \"cb4b1ed5-2192-4877-8be7-ca82af831af5\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-blrrb"
Oct 30 21:51:16.045147 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:16.045121    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb4b1ed5-2192-4877-8be7-ca82af831af5-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-blrrb\" (UID: \"cb4b1ed5-2192-4877-8be7-ca82af831af5\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-blrrb"
Oct 30 21:51:16.045147 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:16.045141    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb4b1ed5-2192-4877-8be7-ca82af831af5-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-blrrb\" (UID: \"cb4b1ed5-2192-4877-8be7-ca82af831af5\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-blrrb"
Oct 30 21:51:16.146602 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:16.146555    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb4b1ed5-2192-4877-8be7-ca82af831af5-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-blrrb\" (UID: \"cb4b1ed5-2192-4877-8be7-ca82af831af5\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-blrrb"
Oct 30 21:51:16.146803 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:16.146638    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb4b1ed5-2192-4877-8be7-ca82af831af5-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-blrrb\" (UID: \"cb4b1ed5-2192-4877-8be7-ca82af831af5\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-blrrb"
Oct 30 21:51:16.146803 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:16.146785    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8ckt\" (UniqueName: \"kubernetes.io/projected/cb4b1ed5-2192-4877-8be7-ca82af831af5-kube-api-access-r8ckt\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-blrrb\" (UID: \"cb4b1ed5-2192-4877-8be7-ca82af831af5\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-blrrb"
Oct 30 21:51:16.146920 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:16.146892    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb4b1ed5-2192-4877-8be7-ca82af831af5-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-blrrb\" (UID: \"cb4b1ed5-2192-4877-8be7-ca82af831af5\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-blrrb"
Oct 30 21:51:16.146995 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:16.146976    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb4b1ed5-2192-4877-8be7-ca82af831af5-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-blrrb\" (UID: \"cb4b1ed5-2192-4877-8be7-ca82af831af5\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-blrrb"
Oct 30 21:51:16.147053 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:16.147007    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb4b1ed5-2192-4877-8be7-ca82af831af5-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-blrrb\" (UID: \"cb4b1ed5-2192-4877-8be7-ca82af831af5\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-blrrb"
Oct 30 21:51:16.147100 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:16.147008    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb4b1ed5-2192-4877-8be7-ca82af831af5-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-blrrb\" (UID: \"cb4b1ed5-2192-4877-8be7-ca82af831af5\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-blrrb"
Oct 30 21:51:16.147258 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:16.147233    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb4b1ed5-2192-4877-8be7-ca82af831af5-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-blrrb\" (UID: \"cb4b1ed5-2192-4877-8be7-ca82af831af5\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-blrrb"
Oct 30 21:51:16.147306 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:16.147266    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb4b1ed5-2192-4877-8be7-ca82af831af5-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-blrrb\" (UID: \"cb4b1ed5-2192-4877-8be7-ca82af831af5\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-blrrb"
Oct 30 21:51:16.148821 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:16.148795    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb4b1ed5-2192-4877-8be7-ca82af831af5-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-blrrb\" (UID: \"cb4b1ed5-2192-4877-8be7-ca82af831af5\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-blrrb"
Oct 30 21:51:16.149209 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:16.149192    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb4b1ed5-2192-4877-8be7-ca82af831af5-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-blrrb\" (UID: \"cb4b1ed5-2192-4877-8be7-ca82af831af5\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-blrrb"
Oct 30 21:51:16.201191 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:16.201123    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8ckt\" (UniqueName: \"kubernetes.io/projected/cb4b1ed5-2192-4877-8be7-ca82af831af5-kube-api-access-r8ckt\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-blrrb\" (UID: \"cb4b1ed5-2192-4877-8be7-ca82af831af5\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-blrrb"
Oct 30 21:51:16.248241 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:16.248207    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-blrrb"
Oct 30 21:51:16.623442 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:16.623410    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-blrrb"]
Oct 30 21:51:16.629453 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:51:16.629418    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcb4b1ed5_2192_4877_8be7_ca82af831af5.slice/crio-e702afeb9a7296db8827fc0f73571b20514dbe2b326af10481f65b187c621d25 WatchSource:0}: Error finding container e702afeb9a7296db8827fc0f73571b20514dbe2b326af10481f65b187c621d25: Status 404 returned error can't find the container with id e702afeb9a7296db8827fc0f73571b20514dbe2b326af10481f65b187c621d25
Oct 30 21:51:17.094884 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:17.094849    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-blrrb" event={"ID":"cb4b1ed5-2192-4877-8be7-ca82af831af5","Type":"ContainerStarted","Data":"e702afeb9a7296db8827fc0f73571b20514dbe2b326af10481f65b187c621d25"}
Oct 30 21:51:18.101344 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:18.101308    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-blrrb" event={"ID":"cb4b1ed5-2192-4877-8be7-ca82af831af5","Type":"ContainerStarted","Data":"2aaa50af029729a10565c7476702530d7f393faf02c43491ecc86a24f64ac49c"}
Oct 30 21:51:18.248618 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:18.248590    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-blrrb"
Oct 30 21:51:19.449701 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:19.449628    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-blrrb" podStartSLOduration=3.843394689 podStartE2EDuration="4.449605579s" totalImagesPullingTime="606.21089ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:51:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:51:18.217508642 +0000 UTC m=+531.830278399" watchObservedRunningTime="2026-10-30 21:51:19.449605579 +0000 UTC m=+533.062375337"
Oct 30 21:51:19.452690 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:19.450195    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zng7g"]
Oct 30 21:51:19.456096 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:19.456074    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zng7g"
Oct 30 21:51:19.553066 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:19.553034    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Oct 30 21:51:19.556411 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:19.556382    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zng7g"]
Oct 30 21:51:19.579075 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:19.579037    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc59d0e7-fbca-4882-8dda-f3f2095c7733-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zng7g\" (UID: \"dc59d0e7-fbca-4882-8dda-f3f2095c7733\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zng7g"
Oct 30 21:51:19.579242 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:19.579101    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc59d0e7-fbca-4882-8dda-f3f2095c7733-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zng7g\" (UID: \"dc59d0e7-fbca-4882-8dda-f3f2095c7733\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zng7g"
Oct 30 21:51:19.579242 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:19.579124    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pwx6\" (UniqueName: \"kubernetes.io/projected/dc59d0e7-fbca-4882-8dda-f3f2095c7733-kube-api-access-9pwx6\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zng7g\" (UID: \"dc59d0e7-fbca-4882-8dda-f3f2095c7733\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zng7g"
Oct 30 21:51:19.579355 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:19.579254    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc59d0e7-fbca-4882-8dda-f3f2095c7733-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zng7g\" (UID: \"dc59d0e7-fbca-4882-8dda-f3f2095c7733\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zng7g"
Oct 30 21:51:19.579355 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:19.579313    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc59d0e7-fbca-4882-8dda-f3f2095c7733-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zng7g\" (UID: \"dc59d0e7-fbca-4882-8dda-f3f2095c7733\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zng7g"
Oct 30 21:51:19.579436 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:19.579416    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc59d0e7-fbca-4882-8dda-f3f2095c7733-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zng7g\" (UID: \"dc59d0e7-fbca-4882-8dda-f3f2095c7733\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zng7g"
Oct 30 21:51:19.680194 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:19.680111    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc59d0e7-fbca-4882-8dda-f3f2095c7733-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zng7g\" (UID: \"dc59d0e7-fbca-4882-8dda-f3f2095c7733\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zng7g"
Oct 30 21:51:19.680194 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:19.680162    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc59d0e7-fbca-4882-8dda-f3f2095c7733-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zng7g\" (UID: \"dc59d0e7-fbca-4882-8dda-f3f2095c7733\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zng7g"
Oct 30 21:51:19.680365 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:19.680202    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc59d0e7-fbca-4882-8dda-f3f2095c7733-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zng7g\" (UID: \"dc59d0e7-fbca-4882-8dda-f3f2095c7733\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zng7g"
Oct 30 21:51:19.680365 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:19.680218    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pwx6\" (UniqueName: \"kubernetes.io/projected/dc59d0e7-fbca-4882-8dda-f3f2095c7733-kube-api-access-9pwx6\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zng7g\" (UID: \"dc59d0e7-fbca-4882-8dda-f3f2095c7733\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zng7g"
Oct 30 21:51:19.680365 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:19.680238    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc59d0e7-fbca-4882-8dda-f3f2095c7733-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zng7g\" (UID: \"dc59d0e7-fbca-4882-8dda-f3f2095c7733\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zng7g"
Oct 30 21:51:19.680465 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:19.680439    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc59d0e7-fbca-4882-8dda-f3f2095c7733-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zng7g\" (UID: \"dc59d0e7-fbca-4882-8dda-f3f2095c7733\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zng7g"
Oct 30 21:51:19.680597 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:19.680572    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc59d0e7-fbca-4882-8dda-f3f2095c7733-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zng7g\" (UID: \"dc59d0e7-fbca-4882-8dda-f3f2095c7733\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zng7g"
Oct 30 21:51:19.680678 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:19.680639    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc59d0e7-fbca-4882-8dda-f3f2095c7733-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zng7g\" (UID: \"dc59d0e7-fbca-4882-8dda-f3f2095c7733\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zng7g"
Oct 30 21:51:19.680825 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:19.680806    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc59d0e7-fbca-4882-8dda-f3f2095c7733-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zng7g\" (UID: \"dc59d0e7-fbca-4882-8dda-f3f2095c7733\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zng7g"
Oct 30 21:51:19.682661 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:19.682639    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc59d0e7-fbca-4882-8dda-f3f2095c7733-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zng7g\" (UID: \"dc59d0e7-fbca-4882-8dda-f3f2095c7733\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zng7g"
Oct 30 21:51:19.682730 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:19.682699    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc59d0e7-fbca-4882-8dda-f3f2095c7733-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zng7g\" (UID: \"dc59d0e7-fbca-4882-8dda-f3f2095c7733\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zng7g"
Oct 30 21:51:19.751049 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:19.751017    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pwx6\" (UniqueName: \"kubernetes.io/projected/dc59d0e7-fbca-4882-8dda-f3f2095c7733-kube-api-access-9pwx6\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zng7g\" (UID: \"dc59d0e7-fbca-4882-8dda-f3f2095c7733\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zng7g"
Oct 30 21:51:19.775355 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:19.775321    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zng7g"
Oct 30 21:51:20.173327 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:20.173292    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zng7g"]
Oct 30 21:51:20.177455 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:51:20.177421    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc59d0e7_fbca_4882_8dda_f3f2095c7733.slice/crio-792c66f0f88658e1c8688b87a8397893d604087d48f0efdb43c99e856381ff66 WatchSource:0}: Error finding container 792c66f0f88658e1c8688b87a8397893d604087d48f0efdb43c99e856381ff66: Status 404 returned error can't find the container with id 792c66f0f88658e1c8688b87a8397893d604087d48f0efdb43c99e856381ff66
Oct 30 21:51:20.211296 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:20.211265    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-hfpzd"
Oct 30 21:51:20.227349 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:20.227317    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-hfpzd"
Oct 30 21:51:21.118180 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:21.118093    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zng7g" event={"ID":"dc59d0e7-fbca-4882-8dda-f3f2095c7733","Type":"ContainerStarted","Data":"8105534f9f89289e061d484ac5c0ae4c2a087662487f75dd80ac56774c73f71b"}
Oct 30 21:51:21.118180 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:21.118147    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zng7g" event={"ID":"dc59d0e7-fbca-4882-8dda-f3f2095c7733","Type":"ContainerStarted","Data":"792c66f0f88658e1c8688b87a8397893d604087d48f0efdb43c99e856381ff66"}
Oct 30 21:51:21.129138 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:21.129117    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-hfpzd"
Oct 30 21:51:21.230160 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:21.230107    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zng7g" podStartSLOduration=1.6009267820000002 podStartE2EDuration="2.230092916s" totalImagesPullingTime="629.166134ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:51:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:51:21.22959012 +0000 UTC m=+534.842359889" watchObservedRunningTime="2026-10-30 21:51:21.230092916 +0000 UTC m=+534.842862671"
Oct 30 21:51:21.776292 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:21.776263    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zng7g"
Oct 30 21:51:24.665841 ip-10-0-130-60 kubenswrapper[5061]: E1030 21:51:24.665797    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 21:51:26.248602 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:26.248561    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-blrrb"
Oct 30 21:51:26.261193 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:26.261167    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-blrrb"
Oct 30 21:51:27.160242 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:27.160206    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-blrrb"
Oct 30 21:51:29.775570 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:29.775518    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zng7g"
Oct 30 21:51:29.787778 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:29.787752    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zng7g"
Oct 30 21:51:30.181690 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:30.181664    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zng7g"
Oct 30 21:51:32.672114 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:32.672081    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vcstl"]
Oct 30 21:51:32.675979 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:32.675959    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vcstl"
Oct 30 21:51:32.711055 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:32.711027    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/877aedb6-e77a-4cfa-9120-2650f6e39604-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-vcstl\" (UID: \"877aedb6-e77a-4cfa-9120-2650f6e39604\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vcstl"
Oct 30 21:51:32.711173 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:32.711060    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/877aedb6-e77a-4cfa-9120-2650f6e39604-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-vcstl\" (UID: \"877aedb6-e77a-4cfa-9120-2650f6e39604\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vcstl"
Oct 30 21:51:32.711173 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:32.711094    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/877aedb6-e77a-4cfa-9120-2650f6e39604-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-vcstl\" (UID: \"877aedb6-e77a-4cfa-9120-2650f6e39604\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vcstl"
Oct 30 21:51:32.711282 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:32.711163    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/877aedb6-e77a-4cfa-9120-2650f6e39604-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-vcstl\" (UID: \"877aedb6-e77a-4cfa-9120-2650f6e39604\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vcstl"
Oct 30 21:51:32.711346 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:32.711326    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42h2w\" (UniqueName: \"kubernetes.io/projected/877aedb6-e77a-4cfa-9120-2650f6e39604-kube-api-access-42h2w\") pod \"maas-default-gateway-openshift-default-694db55c87-vcstl\" (UID: \"877aedb6-e77a-4cfa-9120-2650f6e39604\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vcstl"
Oct 30 21:51:32.711450 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:32.711364    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/877aedb6-e77a-4cfa-9120-2650f6e39604-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-vcstl\" (UID: \"877aedb6-e77a-4cfa-9120-2650f6e39604\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vcstl"
Oct 30 21:51:32.711450 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:32.711424    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/877aedb6-e77a-4cfa-9120-2650f6e39604-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-vcstl\" (UID: \"877aedb6-e77a-4cfa-9120-2650f6e39604\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vcstl"
Oct 30 21:51:32.711565 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:32.711458    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/877aedb6-e77a-4cfa-9120-2650f6e39604-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-vcstl\" (UID: \"877aedb6-e77a-4cfa-9120-2650f6e39604\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vcstl"
Oct 30 21:51:32.711565 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:32.711475    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/877aedb6-e77a-4cfa-9120-2650f6e39604-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-vcstl\" (UID: \"877aedb6-e77a-4cfa-9120-2650f6e39604\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vcstl"
Oct 30 21:51:32.719468 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:32.719442    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-km4g4"]
Oct 30 21:51:32.723035 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:32.723017    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-km4g4"
Oct 30 21:51:32.778219 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:32.778197    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vcstl"]
Oct 30 21:51:32.812188 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:32.812160    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/877aedb6-e77a-4cfa-9120-2650f6e39604-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-vcstl\" (UID: \"877aedb6-e77a-4cfa-9120-2650f6e39604\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vcstl"
Oct 30 21:51:32.812303 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:32.812220    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb4b4d8d-f667-4816-b351-e147ddc48224-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-km4g4\" (UID: \"cb4b4d8d-f667-4816-b351-e147ddc48224\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-km4g4"
Oct 30 21:51:32.812303 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:32.812247    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/877aedb6-e77a-4cfa-9120-2650f6e39604-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-vcstl\" (UID: \"877aedb6-e77a-4cfa-9120-2650f6e39604\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vcstl"
Oct 30 21:51:32.812303 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:32.812277    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/877aedb6-e77a-4cfa-9120-2650f6e39604-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-vcstl\" (UID: \"877aedb6-e77a-4cfa-9120-2650f6e39604\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vcstl"
Oct 30 21:51:32.812431 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:32.812298    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb4b4d8d-f667-4816-b351-e147ddc48224-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-km4g4\" (UID: \"cb4b4d8d-f667-4816-b351-e147ddc48224\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-km4g4"
Oct 30 21:51:32.812431 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:32.812329    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb4b4d8d-f667-4816-b351-e147ddc48224-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-km4g4\" (UID: \"cb4b4d8d-f667-4816-b351-e147ddc48224\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-km4g4"
Oct 30 21:51:32.812518 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:32.812496    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb4b4d8d-f667-4816-b351-e147ddc48224-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-km4g4\" (UID: \"cb4b4d8d-f667-4816-b351-e147ddc48224\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-km4g4"
Oct 30 21:51:32.812597 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:32.812558    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb4b4d8d-f667-4816-b351-e147ddc48224-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-km4g4\" (UID: \"cb4b4d8d-f667-4816-b351-e147ddc48224\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-km4g4"
Oct 30 21:51:32.812654 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:32.812606    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/877aedb6-e77a-4cfa-9120-2650f6e39604-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-vcstl\" (UID: \"877aedb6-e77a-4cfa-9120-2650f6e39604\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vcstl"
Oct 30 21:51:32.812654 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:32.812631    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/877aedb6-e77a-4cfa-9120-2650f6e39604-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-vcstl\" (UID: \"877aedb6-e77a-4cfa-9120-2650f6e39604\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vcstl"
Oct 30 21:51:32.812730 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:32.812670    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/877aedb6-e77a-4cfa-9120-2650f6e39604-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-vcstl\" (UID: \"877aedb6-e77a-4cfa-9120-2650f6e39604\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vcstl"
Oct 30 21:51:32.812730 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:32.812687    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/877aedb6-e77a-4cfa-9120-2650f6e39604-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-vcstl\" (UID: \"877aedb6-e77a-4cfa-9120-2650f6e39604\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vcstl"
Oct 30 21:51:32.812801 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:32.812723    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zsv9\" (UniqueName: \"kubernetes.io/projected/cb4b4d8d-f667-4816-b351-e147ddc48224-kube-api-access-6zsv9\") pod \"maas-default-gateway-openshift-default-694db55c87-km4g4\" (UID: \"cb4b4d8d-f667-4816-b351-e147ddc48224\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-km4g4"
Oct 30 21:51:32.812859 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:32.812845    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42h2w\" (UniqueName: \"kubernetes.io/projected/877aedb6-e77a-4cfa-9120-2650f6e39604-kube-api-access-42h2w\") pod \"maas-default-gateway-openshift-default-694db55c87-vcstl\" (UID: \"877aedb6-e77a-4cfa-9120-2650f6e39604\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vcstl"
Oct 30 21:51:32.812931 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:32.812908    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/877aedb6-e77a-4cfa-9120-2650f6e39604-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-vcstl\" (UID: \"877aedb6-e77a-4cfa-9120-2650f6e39604\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vcstl"
Oct 30 21:51:32.812973 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:32.812919    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/877aedb6-e77a-4cfa-9120-2650f6e39604-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-vcstl\" (UID: \"877aedb6-e77a-4cfa-9120-2650f6e39604\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vcstl"
Oct 30 21:51:32.812973 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:32.812943    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb4b4d8d-f667-4816-b351-e147ddc48224-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-km4g4\" (UID: \"cb4b4d8d-f667-4816-b351-e147ddc48224\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-km4g4"
Oct 30 21:51:32.813053 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:32.812978    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/877aedb6-e77a-4cfa-9120-2650f6e39604-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-vcstl\" (UID: \"877aedb6-e77a-4cfa-9120-2650f6e39604\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vcstl"
Oct 30 21:51:32.813053 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:32.813002    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb4b4d8d-f667-4816-b351-e147ddc48224-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-km4g4\" (UID: \"cb4b4d8d-f667-4816-b351-e147ddc48224\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-km4g4"
Oct 30 21:51:32.813053 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:32.813032    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/877aedb6-e77a-4cfa-9120-2650f6e39604-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-vcstl\" (UID: \"877aedb6-e77a-4cfa-9120-2650f6e39604\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vcstl"
Oct 30 21:51:32.813154 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:32.813100    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb4b4d8d-f667-4816-b351-e147ddc48224-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-km4g4\" (UID: \"cb4b4d8d-f667-4816-b351-e147ddc48224\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-km4g4"
Oct 30 21:51:32.813154 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:32.813125    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/877aedb6-e77a-4cfa-9120-2650f6e39604-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-vcstl\" (UID: \"877aedb6-e77a-4cfa-9120-2650f6e39604\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vcstl"
Oct 30 21:51:32.813246 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:32.813228    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/877aedb6-e77a-4cfa-9120-2650f6e39604-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-vcstl\" (UID: \"877aedb6-e77a-4cfa-9120-2650f6e39604\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vcstl"
Oct 30 21:51:32.815322 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:32.815297    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/877aedb6-e77a-4cfa-9120-2650f6e39604-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-vcstl\" (UID: \"877aedb6-e77a-4cfa-9120-2650f6e39604\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vcstl"
Oct 30 21:51:32.815421 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:32.815333    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/877aedb6-e77a-4cfa-9120-2650f6e39604-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-vcstl\" (UID: \"877aedb6-e77a-4cfa-9120-2650f6e39604\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vcstl"
Oct 30 21:51:32.830447 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:32.830420    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-km4g4"]
Oct 30 21:51:32.867612 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:32.867585    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42h2w\" (UniqueName: \"kubernetes.io/projected/877aedb6-e77a-4cfa-9120-2650f6e39604-kube-api-access-42h2w\") pod \"maas-default-gateway-openshift-default-694db55c87-vcstl\" (UID: \"877aedb6-e77a-4cfa-9120-2650f6e39604\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vcstl"
Oct 30 21:51:32.867727 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:32.867653    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/877aedb6-e77a-4cfa-9120-2650f6e39604-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-vcstl\" (UID: \"877aedb6-e77a-4cfa-9120-2650f6e39604\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vcstl"
Oct 30 21:51:32.913586 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:32.913557    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zsv9\" (UniqueName: \"kubernetes.io/projected/cb4b4d8d-f667-4816-b351-e147ddc48224-kube-api-access-6zsv9\") pod \"maas-default-gateway-openshift-default-694db55c87-km4g4\" (UID: \"cb4b4d8d-f667-4816-b351-e147ddc48224\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-km4g4"
Oct 30 21:51:32.913717 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:32.913611    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb4b4d8d-f667-4816-b351-e147ddc48224-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-km4g4\" (UID: \"cb4b4d8d-f667-4816-b351-e147ddc48224\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-km4g4"
Oct 30 21:51:32.913717 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:32.913639    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb4b4d8d-f667-4816-b351-e147ddc48224-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-km4g4\" (UID: \"cb4b4d8d-f667-4816-b351-e147ddc48224\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-km4g4"
Oct 30 21:51:32.913717 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:32.913686    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb4b4d8d-f667-4816-b351-e147ddc48224-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-km4g4\" (UID: \"cb4b4d8d-f667-4816-b351-e147ddc48224\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-km4g4"
Oct 30 21:51:32.913846 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:32.913741    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb4b4d8d-f667-4816-b351-e147ddc48224-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-km4g4\" (UID: \"cb4b4d8d-f667-4816-b351-e147ddc48224\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-km4g4"
Oct 30 21:51:32.913846 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:32.913775    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb4b4d8d-f667-4816-b351-e147ddc48224-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-km4g4\" (UID: \"cb4b4d8d-f667-4816-b351-e147ddc48224\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-km4g4"
Oct 30 21:51:32.913846 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:32.913800    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb4b4d8d-f667-4816-b351-e147ddc48224-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-km4g4\" (UID: \"cb4b4d8d-f667-4816-b351-e147ddc48224\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-km4g4"
Oct 30 21:51:32.913990 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:32.913849    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb4b4d8d-f667-4816-b351-e147ddc48224-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-km4g4\" (UID: \"cb4b4d8d-f667-4816-b351-e147ddc48224\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-km4g4"
Oct 30 21:51:32.913990 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:32.913874    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb4b4d8d-f667-4816-b351-e147ddc48224-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-km4g4\" (UID: \"cb4b4d8d-f667-4816-b351-e147ddc48224\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-km4g4"
Oct 30 21:51:32.914083 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:32.914049    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb4b4d8d-f667-4816-b351-e147ddc48224-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-km4g4\" (UID: \"cb4b4d8d-f667-4816-b351-e147ddc48224\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-km4g4"
Oct 30 21:51:32.914345 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:32.914317    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb4b4d8d-f667-4816-b351-e147ddc48224-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-km4g4\" (UID: \"cb4b4d8d-f667-4816-b351-e147ddc48224\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-km4g4"
Oct 30 21:51:32.914431 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:32.914353    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb4b4d8d-f667-4816-b351-e147ddc48224-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-km4g4\" (UID: \"cb4b4d8d-f667-4816-b351-e147ddc48224\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-km4g4"
Oct 30 21:51:32.914600 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:32.914516    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb4b4d8d-f667-4816-b351-e147ddc48224-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-km4g4\" (UID: \"cb4b4d8d-f667-4816-b351-e147ddc48224\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-km4g4"
Oct 30 21:51:32.914666 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:32.914635    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb4b4d8d-f667-4816-b351-e147ddc48224-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-km4g4\" (UID: \"cb4b4d8d-f667-4816-b351-e147ddc48224\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-km4g4"
Oct 30 21:51:32.916158 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:32.916140    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb4b4d8d-f667-4816-b351-e147ddc48224-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-km4g4\" (UID: \"cb4b4d8d-f667-4816-b351-e147ddc48224\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-km4g4"
Oct 30 21:51:32.916273 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:32.916255    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb4b4d8d-f667-4816-b351-e147ddc48224-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-km4g4\" (UID: \"cb4b4d8d-f667-4816-b351-e147ddc48224\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-km4g4"
Oct 30 21:51:32.969077 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:32.969010    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb4b4d8d-f667-4816-b351-e147ddc48224-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-km4g4\" (UID: \"cb4b4d8d-f667-4816-b351-e147ddc48224\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-km4g4"
Oct 30 21:51:32.969210 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:32.969192    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zsv9\" (UniqueName: \"kubernetes.io/projected/cb4b4d8d-f667-4816-b351-e147ddc48224-kube-api-access-6zsv9\") pod \"maas-default-gateway-openshift-default-694db55c87-km4g4\" (UID: \"cb4b4d8d-f667-4816-b351-e147ddc48224\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-km4g4"
Oct 30 21:51:32.988131 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:32.988112    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vcstl"
Oct 30 21:51:33.035470 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:33.035439    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-km4g4"
Oct 30 21:51:33.413374 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:33.413347    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vcstl"]
Oct 30 21:51:33.416177 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:33.416156    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-km4g4"]
Oct 30 21:51:33.418487 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:51:33.418455    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod877aedb6_e77a_4cfa_9120_2650f6e39604.slice/crio-6ca341b7f170072dacf1c198443a6f92f35165b12dbf74e0a054d768094de698 WatchSource:0}: Error finding container 6ca341b7f170072dacf1c198443a6f92f35165b12dbf74e0a054d768094de698: Status 404 returned error can't find the container with id 6ca341b7f170072dacf1c198443a6f92f35165b12dbf74e0a054d768094de698
Oct 30 21:51:33.420855 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:33.420824    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:51:33.420934 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:33.420892    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:51:33.420934 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:33.420919    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:51:33.456456 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:51:33.456430    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb4b4d8d_f667_4816_b351_e147ddc48224.slice/crio-486b5bbb2637ba27922a72e3e74b6f137d56765123c443d2e1eca7b364794287 WatchSource:0}: Error finding container 486b5bbb2637ba27922a72e3e74b6f137d56765123c443d2e1eca7b364794287: Status 404 returned error can't find the container with id 486b5bbb2637ba27922a72e3e74b6f137d56765123c443d2e1eca7b364794287
Oct 30 21:51:33.458459 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:33.458432    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:51:33.458569 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:33.458503    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:51:33.458643 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:33.458623    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:51:34.191828 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:34.191789    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vcstl" event={"ID":"877aedb6-e77a-4cfa-9120-2650f6e39604","Type":"ContainerStarted","Data":"8336bb2d481b832628f08f76872ebddde9e71962d6ac9c654192c10e1eb437e1"}
Oct 30 21:51:34.192307 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:34.191849    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vcstl" event={"ID":"877aedb6-e77a-4cfa-9120-2650f6e39604","Type":"ContainerStarted","Data":"6ca341b7f170072dacf1c198443a6f92f35165b12dbf74e0a054d768094de698"}
Oct 30 21:51:34.194100 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:34.194073    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-km4g4" event={"ID":"cb4b4d8d-f667-4816-b351-e147ddc48224","Type":"ContainerStarted","Data":"607adf5c76fe05b67177a64fdd761144fa7b3f8d6f98cff10e60a2127b1bd0a5"}
Oct 30 21:51:34.194222 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:34.194120    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-km4g4" event={"ID":"cb4b4d8d-f667-4816-b351-e147ddc48224","Type":"ContainerStarted","Data":"486b5bbb2637ba27922a72e3e74b6f137d56765123c443d2e1eca7b364794287"}
Oct 30 21:51:34.402667 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:34.402616    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vcstl" podStartSLOduration=2.402601888 podStartE2EDuration="2.402601888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:51:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:51:34.355051564 +0000 UTC m=+547.967821349" watchObservedRunningTime="2026-10-30 21:51:34.402601888 +0000 UTC m=+548.015371646"
Oct 30 21:51:34.473063 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:34.472941    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-km4g4" podStartSLOduration=2.472926123 podStartE2EDuration="2.472926123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:51:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:51:34.471045717 +0000 UTC m=+548.083815474" watchObservedRunningTime="2026-10-30 21:51:34.472926123 +0000 UTC m=+548.085695880"
Oct 30 21:51:34.988706 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:34.988669    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vcstl"
Oct 30 21:51:34.990156 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:34.990130    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-vcstl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body=
Oct 30 21:51:34.990239 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:34.990200    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vcstl" podUID="877aedb6-e77a-4cfa-9120-2650f6e39604" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused"
Oct 30 21:51:35.040552 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:35.037460    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-km4g4"
Oct 30 21:51:35.040552 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:35.037642    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-km4g4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Oct 30 21:51:35.040552 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:35.037699    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-km4g4" podUID="cb4b4d8d-f667-4816-b351-e147ddc48224" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Oct 30 21:51:35.988646 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:35.988604    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-vcstl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body=
Oct 30 21:51:35.989126 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:35.988658    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vcstl" podUID="877aedb6-e77a-4cfa-9120-2650f6e39604" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused"
Oct 30 21:51:36.038893 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:36.038857    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-km4g4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Oct 30 21:51:36.039131 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:36.039102    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-km4g4" podUID="cb4b4d8d-f667-4816-b351-e147ddc48224" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Oct 30 21:51:36.993551 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:36.992911    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-vcstl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body=
Oct 30 21:51:36.993551 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:36.992973    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vcstl" podUID="877aedb6-e77a-4cfa-9120-2650f6e39604" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused"
Oct 30 21:51:37.039208 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:37.039171    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-km4g4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Oct 30 21:51:37.039467 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:37.039439    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-km4g4" podUID="cb4b4d8d-f667-4816-b351-e147ddc48224" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Oct 30 21:51:37.997967 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:37.997933    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-vcstl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body=
Oct 30 21:51:37.998616 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:37.998581    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vcstl" podUID="877aedb6-e77a-4cfa-9120-2650f6e39604" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused"
Oct 30 21:51:38.039822 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:38.039785    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-km4g4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Oct 30 21:51:38.040081 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:38.040043    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-km4g4" podUID="cb4b4d8d-f667-4816-b351-e147ddc48224" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Oct 30 21:51:38.988986 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:38.988950    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-vcstl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body=
Oct 30 21:51:38.989137 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:38.989001    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vcstl" podUID="877aedb6-e77a-4cfa-9120-2650f6e39604" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused"
Oct 30 21:51:39.035932 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:39.035894    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-km4g4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Oct 30 21:51:39.036482 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:39.036449    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-km4g4" podUID="cb4b4d8d-f667-4816-b351-e147ddc48224" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Oct 30 21:51:39.989022 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:39.988973    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-vcstl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body=
Oct 30 21:51:39.989229 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:39.989053    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vcstl" podUID="877aedb6-e77a-4cfa-9120-2650f6e39604" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused"
Oct 30 21:51:40.036356 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:40.036311    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-km4g4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Oct 30 21:51:40.036862 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:40.036375    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-km4g4" podUID="cb4b4d8d-f667-4816-b351-e147ddc48224" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Oct 30 21:51:41.988680 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:41.988643    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-vcstl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:51:41.989174 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:41.988697    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vcstl" podUID="877aedb6-e77a-4cfa-9120-2650f6e39604" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:51:42.036539 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:42.036483    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-km4g4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:51:42.036669 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:42.036568    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-km4g4" podUID="cb4b4d8d-f667-4816-b351-e147ddc48224" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:51:43.029741 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:43.029687    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-vcstl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:51:43.030400 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:43.030363    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vcstl" podUID="877aedb6-e77a-4cfa-9120-2650f6e39604" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:51:43.077656 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:43.077624    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-km4g4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:51:43.077808 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:43.077680    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-km4g4" podUID="cb4b4d8d-f667-4816-b351-e147ddc48224" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:51:43.725993 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:43.725958    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vcstl"
Oct 30 21:51:43.726386 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:43.726363    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vcstl"
Oct 30 21:51:43.727190 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:43.727168    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vcstl"
Oct 30 21:51:43.937667 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:43.937636    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-km4g4"
Oct 30 21:51:43.937993 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:43.937967    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-km4g4"
Oct 30 21:51:43.938847 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:43.938822    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-km4g4"
Oct 30 21:51:45.561818 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:45.561787    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-f7b7d88f4-65wcn"]
Oct 30 21:51:45.565866 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:45.565843    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-f7b7d88f4-65wcn"
Oct 30 21:51:45.624960 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:45.624929    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-f7b7d88f4-65wcn"]
Oct 30 21:51:45.627704 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:45.627374    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ffw7\" (UniqueName: \"kubernetes.io/projected/2c715ba0-1ad7-4ae9-a88f-9261fed32e5b-kube-api-access-5ffw7\") pod \"authorino-f7b7d88f4-65wcn\" (UID: \"2c715ba0-1ad7-4ae9-a88f-9261fed32e5b\") " pod="kuadrant-system/authorino-f7b7d88f4-65wcn"
Oct 30 21:51:45.627704 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:45.627422    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2c715ba0-1ad7-4ae9-a88f-9261fed32e5b-tls-cert\") pod \"authorino-f7b7d88f4-65wcn\" (UID: \"2c715ba0-1ad7-4ae9-a88f-9261fed32e5b\") " pod="kuadrant-system/authorino-f7b7d88f4-65wcn"
Oct 30 21:51:45.728301 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:45.728266    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ffw7\" (UniqueName: \"kubernetes.io/projected/2c715ba0-1ad7-4ae9-a88f-9261fed32e5b-kube-api-access-5ffw7\") pod \"authorino-f7b7d88f4-65wcn\" (UID: \"2c715ba0-1ad7-4ae9-a88f-9261fed32e5b\") " pod="kuadrant-system/authorino-f7b7d88f4-65wcn"
Oct 30 21:51:45.728424 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:45.728317    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2c715ba0-1ad7-4ae9-a88f-9261fed32e5b-tls-cert\") pod \"authorino-f7b7d88f4-65wcn\" (UID: \"2c715ba0-1ad7-4ae9-a88f-9261fed32e5b\") " pod="kuadrant-system/authorino-f7b7d88f4-65wcn"
Oct 30 21:51:45.730822 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:45.730796    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2c715ba0-1ad7-4ae9-a88f-9261fed32e5b-tls-cert\") pod \"authorino-f7b7d88f4-65wcn\" (UID: \"2c715ba0-1ad7-4ae9-a88f-9261fed32e5b\") " pod="kuadrant-system/authorino-f7b7d88f4-65wcn"
Oct 30 21:51:45.784775 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:45.784749    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ffw7\" (UniqueName: \"kubernetes.io/projected/2c715ba0-1ad7-4ae9-a88f-9261fed32e5b-kube-api-access-5ffw7\") pod \"authorino-f7b7d88f4-65wcn\" (UID: \"2c715ba0-1ad7-4ae9-a88f-9261fed32e5b\") " pod="kuadrant-system/authorino-f7b7d88f4-65wcn"
Oct 30 21:51:45.877702 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:45.877627    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-f7b7d88f4-65wcn"
Oct 30 21:51:46.247257 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:46.247182    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-f7b7d88f4-65wcn"]
Oct 30 21:51:46.252509 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:51:46.252484    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2c715ba0_1ad7_4ae9_a88f_9261fed32e5b.slice/crio-e8fc182e8294ffe9858ce66ec148f1467c8317b006a0f008c038198bc9223286 WatchSource:0}: Error finding container e8fc182e8294ffe9858ce66ec148f1467c8317b006a0f008c038198bc9223286: Status 404 returned error can't find the container with id e8fc182e8294ffe9858ce66ec148f1467c8317b006a0f008c038198bc9223286
Oct 30 21:51:46.314150 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:46.314116    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-f7b7d88f4-65wcn" event={"ID":"2c715ba0-1ad7-4ae9-a88f-9261fed32e5b","Type":"ContainerStarted","Data":"e8fc182e8294ffe9858ce66ec148f1467c8317b006a0f008c038198bc9223286"}
Oct 30 21:51:48.325863 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:48.325827    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-f7b7d88f4-65wcn" event={"ID":"2c715ba0-1ad7-4ae9-a88f-9261fed32e5b","Type":"ContainerStarted","Data":"ee2e82fc4526f41460edcab5796b9f85d978a930f3bb4b0aac7dadbd7c4d43d3"}
Oct 30 21:51:48.435301 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:48.435249    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-f7b7d88f4-65wcn" podStartSLOduration=1.675921787 podStartE2EDuration="3.435236868s" totalImagesPullingTime="1.759315081s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:51:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:51:48.434305021 +0000 UTC m=+562.047074778" watchObservedRunningTime="2026-10-30 21:51:48.435236868 +0000 UTC m=+562.048006626"
Oct 30 21:51:48.483422 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:48.483390    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-96689fcc7-ttkdx"]
Oct 30 21:51:48.483642 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:48.483603    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-96689fcc7-ttkdx" podUID="1831b1cd-70a7-4004-8fc7-0949179c0292" containerName="authorino" containerID="cri-o://3505d85a612ca508bd381506d93ee2af1d9340c45e85c96cb71bf29a7543d266" gracePeriod=30
Oct 30 21:51:48.750015 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:48.749991    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-96689fcc7-ttkdx"
Oct 30 21:51:48.857824 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:48.857745    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1831b1cd-70a7-4004-8fc7-0949179c0292-tls-cert\" (UniqueName: \"kubernetes.io/secret/1831b1cd-70a7-4004-8fc7-0949179c0292-tls-cert\") pod \"1831b1cd-70a7-4004-8fc7-0949179c0292\" (UID: \"1831b1cd-70a7-4004-8fc7-0949179c0292\") "
Oct 30 21:51:48.857824 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:48.857822    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1831b1cd-70a7-4004-8fc7-0949179c0292-kube-api-access-rsxs5\" (UniqueName: \"kubernetes.io/projected/1831b1cd-70a7-4004-8fc7-0949179c0292-kube-api-access-rsxs5\") pod \"1831b1cd-70a7-4004-8fc7-0949179c0292\" (UID: \"1831b1cd-70a7-4004-8fc7-0949179c0292\") "
Oct 30 21:51:48.859947 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:48.859915    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1831b1cd-70a7-4004-8fc7-0949179c0292-kube-api-access-rsxs5" pod "1831b1cd-70a7-4004-8fc7-0949179c0292" (UID: "1831b1cd-70a7-4004-8fc7-0949179c0292"). InnerVolumeSpecName "kube-api-access-rsxs5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:51:48.878576 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:48.878506    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1831b1cd-70a7-4004-8fc7-0949179c0292-tls-cert" pod "1831b1cd-70a7-4004-8fc7-0949179c0292" (UID: "1831b1cd-70a7-4004-8fc7-0949179c0292"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:51:48.959066 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:48.959024    5061 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1831b1cd-70a7-4004-8fc7-0949179c0292-tls-cert\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:51:48.959066 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:48.959067    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rsxs5\" (UniqueName: \"kubernetes.io/projected/1831b1cd-70a7-4004-8fc7-0949179c0292-kube-api-access-rsxs5\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:51:49.332159 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:49.332124    5061 generic.go:356] "Generic (PLEG): container finished" podID="1831b1cd-70a7-4004-8fc7-0949179c0292" containerID="3505d85a612ca508bd381506d93ee2af1d9340c45e85c96cb71bf29a7543d266" exitCode=0
Oct 30 21:51:49.332661 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:49.332166    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-96689fcc7-ttkdx"
Oct 30 21:51:49.332661 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:49.332215    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-96689fcc7-ttkdx" event={"ID":"1831b1cd-70a7-4004-8fc7-0949179c0292","Type":"ContainerDied","Data":"3505d85a612ca508bd381506d93ee2af1d9340c45e85c96cb71bf29a7543d266"}
Oct 30 21:51:49.332661 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:49.332253    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-96689fcc7-ttkdx" event={"ID":"1831b1cd-70a7-4004-8fc7-0949179c0292","Type":"ContainerDied","Data":"f83e12ecd1a83dfd1c2ce7d63785fc1fb66fa7f4af4743084d98b66a47631c88"}
Oct 30 21:51:49.332661 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:49.332269    5061 scope.go:122] "RemoveContainer" containerID="3505d85a612ca508bd381506d93ee2af1d9340c45e85c96cb71bf29a7543d266"
Oct 30 21:51:49.343055 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:49.343037    5061 scope.go:122] "RemoveContainer" containerID="3505d85a612ca508bd381506d93ee2af1d9340c45e85c96cb71bf29a7543d266"
Oct 30 21:51:49.343287 ip-10-0-130-60 kubenswrapper[5061]: E1030 21:51:49.343270    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3505d85a612ca508bd381506d93ee2af1d9340c45e85c96cb71bf29a7543d266\": container with ID starting with 3505d85a612ca508bd381506d93ee2af1d9340c45e85c96cb71bf29a7543d266 not found: ID does not exist" containerID="3505d85a612ca508bd381506d93ee2af1d9340c45e85c96cb71bf29a7543d266"
Oct 30 21:51:49.343334 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:49.343301    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3505d85a612ca508bd381506d93ee2af1d9340c45e85c96cb71bf29a7543d266"} err="failed to get container status \"3505d85a612ca508bd381506d93ee2af1d9340c45e85c96cb71bf29a7543d266\": rpc error: code = NotFound desc = could not find container \"3505d85a612ca508bd381506d93ee2af1d9340c45e85c96cb71bf29a7543d266\": container with ID starting with 3505d85a612ca508bd381506d93ee2af1d9340c45e85c96cb71bf29a7543d266 not found: ID does not exist"
Oct 30 21:51:49.489039 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:49.488999    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-96689fcc7-ttkdx"]
Oct 30 21:51:49.536422 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:49.536383    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-96689fcc7-ttkdx"]
Oct 30 21:51:50.527461 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:50.527427    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1831b1cd-70a7-4004-8fc7-0949179c0292" path="/var/lib/kubelet/pods/1831b1cd-70a7-4004-8fc7-0949179c0292/volumes"
Oct 30 21:51:54.665996 ip-10-0-130-60 kubenswrapper[5061]: E1030 21:51:54.665963    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 21:51:56.151377 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:56.151297    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-7bb9489676-cxzp4"]
Oct 30 21:51:56.151973 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:56.151948    5061 cpu_manager.go:414] "containerMap: removing container" podUID="1831b1cd-70a7-4004-8fc7-0949179c0292" containerName="authorino"
Oct 30 21:51:56.151973 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:56.151974    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1831b1cd-70a7-4004-8fc7-0949179c0292" containerName="authorino"
Oct 30 21:51:56.152173 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:56.152091    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="1831b1cd-70a7-4004-8fc7-0949179c0292" containerName="authorino"
Oct 30 21:51:56.158639 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:56.158616    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-7bb9489676-cxzp4"
Oct 30 21:51:56.235659 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:56.235626    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pql96\" (UniqueName: \"kubernetes.io/projected/8867dea6-7639-4fca-95d7-ee7789e21a1d-kube-api-access-pql96\") pod \"payload-processing-7bb9489676-cxzp4\" (UID: \"8867dea6-7639-4fca-95d7-ee7789e21a1d\") " pod="openshift-ingress/payload-processing-7bb9489676-cxzp4"
Oct 30 21:51:56.235849 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:56.235669    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8867dea6-7639-4fca-95d7-ee7789e21a1d-plugins-config-volume\") pod \"payload-processing-7bb9489676-cxzp4\" (UID: \"8867dea6-7639-4fca-95d7-ee7789e21a1d\") " pod="openshift-ingress/payload-processing-7bb9489676-cxzp4"
Oct 30 21:51:56.257212 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:56.257185    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Oct 30 21:51:56.257372 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:56.257221    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-26prj\""
Oct 30 21:51:56.259382 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:56.259358    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-7bb9489676-cxzp4"]
Oct 30 21:51:56.336933 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:56.336910    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pql96\" (UniqueName: \"kubernetes.io/projected/8867dea6-7639-4fca-95d7-ee7789e21a1d-kube-api-access-pql96\") pod \"payload-processing-7bb9489676-cxzp4\" (UID: \"8867dea6-7639-4fca-95d7-ee7789e21a1d\") " pod="openshift-ingress/payload-processing-7bb9489676-cxzp4"
Oct 30 21:51:56.337057 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:56.336947    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8867dea6-7639-4fca-95d7-ee7789e21a1d-plugins-config-volume\") pod \"payload-processing-7bb9489676-cxzp4\" (UID: \"8867dea6-7639-4fca-95d7-ee7789e21a1d\") " pod="openshift-ingress/payload-processing-7bb9489676-cxzp4"
Oct 30 21:51:56.337667 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:56.337650    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8867dea6-7639-4fca-95d7-ee7789e21a1d-plugins-config-volume\") pod \"payload-processing-7bb9489676-cxzp4\" (UID: \"8867dea6-7639-4fca-95d7-ee7789e21a1d\") " pod="openshift-ingress/payload-processing-7bb9489676-cxzp4"
Oct 30 21:51:56.391725 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:56.391700    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pql96\" (UniqueName: \"kubernetes.io/projected/8867dea6-7639-4fca-95d7-ee7789e21a1d-kube-api-access-pql96\") pod \"payload-processing-7bb9489676-cxzp4\" (UID: \"8867dea6-7639-4fca-95d7-ee7789e21a1d\") " pod="openshift-ingress/payload-processing-7bb9489676-cxzp4"
Oct 30 21:51:56.471873 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:56.471809    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-7bb9489676-cxzp4"
Oct 30 21:51:56.848243 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:56.843154    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-7bb9489676-cxzp4"]
Oct 30 21:51:56.850216 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:51:56.850182    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8867dea6_7639_4fca_95d7_ee7789e21a1d.slice/crio-16e1897ff39d6210e99c943dfecd45f4a676bf2448596a69ea6faed82f116d4c WatchSource:0}: Error finding container 16e1897ff39d6210e99c943dfecd45f4a676bf2448596a69ea6faed82f116d4c: Status 404 returned error can't find the container with id 16e1897ff39d6210e99c943dfecd45f4a676bf2448596a69ea6faed82f116d4c
Oct 30 21:51:57.373967 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:51:57.373927    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-7bb9489676-cxzp4" event={"ID":"8867dea6-7639-4fca-95d7-ee7789e21a1d","Type":"ContainerStarted","Data":"16e1897ff39d6210e99c943dfecd45f4a676bf2448596a69ea6faed82f116d4c"}
Oct 30 21:52:00.390663 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:00.390629    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-7bb9489676-cxzp4" event={"ID":"8867dea6-7639-4fca-95d7-ee7789e21a1d","Type":"ContainerStarted","Data":"28a9742eb2adf069730256074ba90c2fdaea40aba39c347bd65a463396ee26a6"}
Oct 30 21:52:00.391184 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:00.390768    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-7bb9489676-cxzp4"
Oct 30 21:52:00.500891 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:00.500849    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-7bb9489676-cxzp4" podStartSLOduration=1.752415007 podStartE2EDuration="4.500837432s" totalImagesPullingTime="2.748422425s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:51:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:52:00.500308403 +0000 UTC m=+574.113078160" watchObservedRunningTime="2026-10-30 21:52:00.500837432 +0000 UTC m=+574.113607189"
Oct 30 21:52:11.399223 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:11.399138    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-7bb9489676-cxzp4"
Oct 30 21:52:24.666595 ip-10-0-130-60 kubenswrapper[5061]: E1030 21:52:24.666555    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 21:52:26.591940 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:26.591909    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-sclrt_5e74071e-0462-42b5-a05c-d90bb638478a/console-operator/2.log"
Oct 30 21:52:26.593040 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:26.593012    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-sclrt_5e74071e-0462-42b5-a05c-d90bb638478a/console-operator/2.log"
Oct 30 21:52:49.512591 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:49.512554    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-chgcq"]
Oct 30 21:52:49.516670 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:49.516649    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-chgcq"
Oct 30 21:52:49.595917 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:49.595884    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-chgcq"]
Oct 30 21:52:49.654678 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:49.654653    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-f776cb-openshift-default-dockercfg-gj5q6\""
Oct 30 21:52:49.659219 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:49.659191    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/147f4f98-ac6a-4b58-8750-a8410dd6324e-istio-data\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-chgcq\" (UID: \"147f4f98-ac6a-4b58-8750-a8410dd6324e\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-chgcq"
Oct 30 21:52:49.659356 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:49.659229    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/147f4f98-ac6a-4b58-8750-a8410dd6324e-istio-token\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-chgcq\" (UID: \"147f4f98-ac6a-4b58-8750-a8410dd6324e\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-chgcq"
Oct 30 21:52:49.659451 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:49.659428    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/147f4f98-ac6a-4b58-8750-a8410dd6324e-istiod-ca-cert\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-chgcq\" (UID: \"147f4f98-ac6a-4b58-8750-a8410dd6324e\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-chgcq"
Oct 30 21:52:49.659494 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:49.659471    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/147f4f98-ac6a-4b58-8750-a8410dd6324e-istio-podinfo\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-chgcq\" (UID: \"147f4f98-ac6a-4b58-8750-a8410dd6324e\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-chgcq"
Oct 30 21:52:49.659568 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:49.659506    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zblzp\" (UniqueName: \"kubernetes.io/projected/147f4f98-ac6a-4b58-8750-a8410dd6324e-kube-api-access-zblzp\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-chgcq\" (UID: \"147f4f98-ac6a-4b58-8750-a8410dd6324e\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-chgcq"
Oct 30 21:52:49.659633 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:49.659602    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/147f4f98-ac6a-4b58-8750-a8410dd6324e-workload-socket\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-chgcq\" (UID: \"147f4f98-ac6a-4b58-8750-a8410dd6324e\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-chgcq"
Oct 30 21:52:49.659713 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:49.659696    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/147f4f98-ac6a-4b58-8750-a8410dd6324e-credential-socket\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-chgcq\" (UID: \"147f4f98-ac6a-4b58-8750-a8410dd6324e\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-chgcq"
Oct 30 21:52:49.659770 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:49.659734    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/147f4f98-ac6a-4b58-8750-a8410dd6324e-istio-envoy\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-chgcq\" (UID: \"147f4f98-ac6a-4b58-8750-a8410dd6324e\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-chgcq"
Oct 30 21:52:49.659869 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:49.659845    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/147f4f98-ac6a-4b58-8750-a8410dd6324e-workload-certs\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-chgcq\" (UID: \"147f4f98-ac6a-4b58-8750-a8410dd6324e\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-chgcq"
Oct 30 21:52:49.760992 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:49.760953    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/147f4f98-ac6a-4b58-8750-a8410dd6324e-istiod-ca-cert\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-chgcq\" (UID: \"147f4f98-ac6a-4b58-8750-a8410dd6324e\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-chgcq"
Oct 30 21:52:49.760992 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:49.760991    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/147f4f98-ac6a-4b58-8750-a8410dd6324e-istio-podinfo\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-chgcq\" (UID: \"147f4f98-ac6a-4b58-8750-a8410dd6324e\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-chgcq"
Oct 30 21:52:49.761242 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:49.761013    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zblzp\" (UniqueName: \"kubernetes.io/projected/147f4f98-ac6a-4b58-8750-a8410dd6324e-kube-api-access-zblzp\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-chgcq\" (UID: \"147f4f98-ac6a-4b58-8750-a8410dd6324e\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-chgcq"
Oct 30 21:52:49.761242 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:49.761038    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/147f4f98-ac6a-4b58-8750-a8410dd6324e-workload-socket\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-chgcq\" (UID: \"147f4f98-ac6a-4b58-8750-a8410dd6324e\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-chgcq"
Oct 30 21:52:49.761242 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:49.761069    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/147f4f98-ac6a-4b58-8750-a8410dd6324e-credential-socket\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-chgcq\" (UID: \"147f4f98-ac6a-4b58-8750-a8410dd6324e\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-chgcq"
Oct 30 21:52:49.761242 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:49.761088    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/147f4f98-ac6a-4b58-8750-a8410dd6324e-istio-envoy\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-chgcq\" (UID: \"147f4f98-ac6a-4b58-8750-a8410dd6324e\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-chgcq"
Oct 30 21:52:49.761242 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:49.761117    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/147f4f98-ac6a-4b58-8750-a8410dd6324e-workload-certs\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-chgcq\" (UID: \"147f4f98-ac6a-4b58-8750-a8410dd6324e\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-chgcq"
Oct 30 21:52:49.761242 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:49.761182    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/147f4f98-ac6a-4b58-8750-a8410dd6324e-istio-data\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-chgcq\" (UID: \"147f4f98-ac6a-4b58-8750-a8410dd6324e\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-chgcq"
Oct 30 21:52:49.761242 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:49.761202    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/147f4f98-ac6a-4b58-8750-a8410dd6324e-istio-token\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-chgcq\" (UID: \"147f4f98-ac6a-4b58-8750-a8410dd6324e\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-chgcq"
Oct 30 21:52:49.761644 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:49.761496    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/147f4f98-ac6a-4b58-8750-a8410dd6324e-workload-socket\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-chgcq\" (UID: \"147f4f98-ac6a-4b58-8750-a8410dd6324e\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-chgcq"
Oct 30 21:52:49.761644 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:49.761622    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/147f4f98-ac6a-4b58-8750-a8410dd6324e-credential-socket\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-chgcq\" (UID: \"147f4f98-ac6a-4b58-8750-a8410dd6324e\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-chgcq"
Oct 30 21:52:49.761724 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:49.761674    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/147f4f98-ac6a-4b58-8750-a8410dd6324e-istio-data\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-chgcq\" (UID: \"147f4f98-ac6a-4b58-8750-a8410dd6324e\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-chgcq"
Oct 30 21:52:49.761724 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:49.761676    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/147f4f98-ac6a-4b58-8750-a8410dd6324e-workload-certs\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-chgcq\" (UID: \"147f4f98-ac6a-4b58-8750-a8410dd6324e\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-chgcq"
Oct 30 21:52:49.762110 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:49.762089    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/147f4f98-ac6a-4b58-8750-a8410dd6324e-istiod-ca-cert\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-chgcq\" (UID: \"147f4f98-ac6a-4b58-8750-a8410dd6324e\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-chgcq"
Oct 30 21:52:49.763973 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:49.763916    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/147f4f98-ac6a-4b58-8750-a8410dd6324e-istio-envoy\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-chgcq\" (UID: \"147f4f98-ac6a-4b58-8750-a8410dd6324e\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-chgcq"
Oct 30 21:52:49.764188 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:49.764169    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/147f4f98-ac6a-4b58-8750-a8410dd6324e-istio-podinfo\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-chgcq\" (UID: \"147f4f98-ac6a-4b58-8750-a8410dd6324e\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-chgcq"
Oct 30 21:52:49.831176 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:49.831143    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/147f4f98-ac6a-4b58-8750-a8410dd6324e-istio-token\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-chgcq\" (UID: \"147f4f98-ac6a-4b58-8750-a8410dd6324e\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-chgcq"
Oct 30 21:52:49.831362 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:49.831337    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zblzp\" (UniqueName: \"kubernetes.io/projected/147f4f98-ac6a-4b58-8750-a8410dd6324e-kube-api-access-zblzp\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-chgcq\" (UID: \"147f4f98-ac6a-4b58-8750-a8410dd6324e\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-chgcq"
Oct 30 21:52:49.895921 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:49.895894    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-ddh7p"]
Oct 30 21:52:49.901046 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:49.901023    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-ddh7p"
Oct 30 21:52:49.950926 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:49.950877    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-17d809-openshift-default-dockercfg-ppc95\""
Oct 30 21:52:49.960085 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:49.960058    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-ddh7p"]
Oct 30 21:52:49.963499 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:49.963472    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba3aed70-33f3-4f0f-87cd-a8c19c347cd8-istio-data\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-ddh7p\" (UID: \"ba3aed70-33f3-4f0f-87cd-a8c19c347cd8\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-ddh7p"
Oct 30 21:52:49.963668 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:49.963562    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba3aed70-33f3-4f0f-87cd-a8c19c347cd8-workload-certs\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-ddh7p\" (UID: \"ba3aed70-33f3-4f0f-87cd-a8c19c347cd8\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-ddh7p"
Oct 30 21:52:49.963668 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:49.963594    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba3aed70-33f3-4f0f-87cd-a8c19c347cd8-istiod-ca-cert\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-ddh7p\" (UID: \"ba3aed70-33f3-4f0f-87cd-a8c19c347cd8\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-ddh7p"
Oct 30 21:52:49.963668 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:49.963660    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba3aed70-33f3-4f0f-87cd-a8c19c347cd8-credential-socket\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-ddh7p\" (UID: \"ba3aed70-33f3-4f0f-87cd-a8c19c347cd8\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-ddh7p"
Oct 30 21:52:49.963831 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:49.963715    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba3aed70-33f3-4f0f-87cd-a8c19c347cd8-istio-podinfo\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-ddh7p\" (UID: \"ba3aed70-33f3-4f0f-87cd-a8c19c347cd8\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-ddh7p"
Oct 30 21:52:49.963831 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:49.963769    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba3aed70-33f3-4f0f-87cd-a8c19c347cd8-istio-token\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-ddh7p\" (UID: \"ba3aed70-33f3-4f0f-87cd-a8c19c347cd8\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-ddh7p"
Oct 30 21:52:49.963831 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:49.963799    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba3aed70-33f3-4f0f-87cd-a8c19c347cd8-istio-envoy\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-ddh7p\" (UID: \"ba3aed70-33f3-4f0f-87cd-a8c19c347cd8\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-ddh7p"
Oct 30 21:52:49.963981 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:49.963862    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jnbl\" (UniqueName: \"kubernetes.io/projected/ba3aed70-33f3-4f0f-87cd-a8c19c347cd8-kube-api-access-7jnbl\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-ddh7p\" (UID: \"ba3aed70-33f3-4f0f-87cd-a8c19c347cd8\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-ddh7p"
Oct 30 21:52:49.963981 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:49.963905    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba3aed70-33f3-4f0f-87cd-a8c19c347cd8-workload-socket\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-ddh7p\" (UID: \"ba3aed70-33f3-4f0f-87cd-a8c19c347cd8\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-ddh7p"
Oct 30 21:52:50.064450 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:50.064417    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba3aed70-33f3-4f0f-87cd-a8c19c347cd8-workload-socket\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-ddh7p\" (UID: \"ba3aed70-33f3-4f0f-87cd-a8c19c347cd8\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-ddh7p"
Oct 30 21:52:50.064684 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:50.064465    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba3aed70-33f3-4f0f-87cd-a8c19c347cd8-istio-data\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-ddh7p\" (UID: \"ba3aed70-33f3-4f0f-87cd-a8c19c347cd8\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-ddh7p"
Oct 30 21:52:50.064684 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:50.064509    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba3aed70-33f3-4f0f-87cd-a8c19c347cd8-workload-certs\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-ddh7p\" (UID: \"ba3aed70-33f3-4f0f-87cd-a8c19c347cd8\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-ddh7p"
Oct 30 21:52:50.064684 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:50.064558    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba3aed70-33f3-4f0f-87cd-a8c19c347cd8-istiod-ca-cert\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-ddh7p\" (UID: \"ba3aed70-33f3-4f0f-87cd-a8c19c347cd8\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-ddh7p"
Oct 30 21:52:50.064684 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:50.064627    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba3aed70-33f3-4f0f-87cd-a8c19c347cd8-credential-socket\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-ddh7p\" (UID: \"ba3aed70-33f3-4f0f-87cd-a8c19c347cd8\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-ddh7p"
Oct 30 21:52:50.064684 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:50.064682    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba3aed70-33f3-4f0f-87cd-a8c19c347cd8-istio-podinfo\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-ddh7p\" (UID: \"ba3aed70-33f3-4f0f-87cd-a8c19c347cd8\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-ddh7p"
Oct 30 21:52:50.065021 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:50.064737    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba3aed70-33f3-4f0f-87cd-a8c19c347cd8-istio-token\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-ddh7p\" (UID: \"ba3aed70-33f3-4f0f-87cd-a8c19c347cd8\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-ddh7p"
Oct 30 21:52:50.065021 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:50.064771    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba3aed70-33f3-4f0f-87cd-a8c19c347cd8-istio-envoy\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-ddh7p\" (UID: \"ba3aed70-33f3-4f0f-87cd-a8c19c347cd8\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-ddh7p"
Oct 30 21:52:50.065021 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:50.064829    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jnbl\" (UniqueName: \"kubernetes.io/projected/ba3aed70-33f3-4f0f-87cd-a8c19c347cd8-kube-api-access-7jnbl\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-ddh7p\" (UID: \"ba3aed70-33f3-4f0f-87cd-a8c19c347cd8\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-ddh7p"
Oct 30 21:52:50.065021 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:50.064904    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba3aed70-33f3-4f0f-87cd-a8c19c347cd8-workload-socket\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-ddh7p\" (UID: \"ba3aed70-33f3-4f0f-87cd-a8c19c347cd8\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-ddh7p"
Oct 30 21:52:50.065246 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:50.065221    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba3aed70-33f3-4f0f-87cd-a8c19c347cd8-workload-certs\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-ddh7p\" (UID: \"ba3aed70-33f3-4f0f-87cd-a8c19c347cd8\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-ddh7p"
Oct 30 21:52:50.065301 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:50.065272    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba3aed70-33f3-4f0f-87cd-a8c19c347cd8-istio-data\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-ddh7p\" (UID: \"ba3aed70-33f3-4f0f-87cd-a8c19c347cd8\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-ddh7p"
Oct 30 21:52:50.065728 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:50.065682    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba3aed70-33f3-4f0f-87cd-a8c19c347cd8-credential-socket\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-ddh7p\" (UID: \"ba3aed70-33f3-4f0f-87cd-a8c19c347cd8\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-ddh7p"
Oct 30 21:52:50.068357 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:50.068330    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba3aed70-33f3-4f0f-87cd-a8c19c347cd8-istio-podinfo\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-ddh7p\" (UID: \"ba3aed70-33f3-4f0f-87cd-a8c19c347cd8\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-ddh7p"
Oct 30 21:52:50.068628 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:50.068601    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba3aed70-33f3-4f0f-87cd-a8c19c347cd8-istiod-ca-cert\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-ddh7p\" (UID: \"ba3aed70-33f3-4f0f-87cd-a8c19c347cd8\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-ddh7p"
Oct 30 21:52:50.069105 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:50.069018    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba3aed70-33f3-4f0f-87cd-a8c19c347cd8-istio-envoy\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-ddh7p\" (UID: \"ba3aed70-33f3-4f0f-87cd-a8c19c347cd8\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-ddh7p"
Oct 30 21:52:50.119833 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:50.119813    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba3aed70-33f3-4f0f-87cd-a8c19c347cd8-istio-token\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-ddh7p\" (UID: \"ba3aed70-33f3-4f0f-87cd-a8c19c347cd8\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-ddh7p"
Oct 30 21:52:50.120114 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:50.120093    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jnbl\" (UniqueName: \"kubernetes.io/projected/ba3aed70-33f3-4f0f-87cd-a8c19c347cd8-kube-api-access-7jnbl\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-ddh7p\" (UID: \"ba3aed70-33f3-4f0f-87cd-a8c19c347cd8\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-ddh7p"
Oct 30 21:52:50.128859 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:50.128840    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-chgcq"
Oct 30 21:52:50.215201 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:50.215087    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-ddh7p"
Oct 30 21:52:50.570791 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:50.570764    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-zskrq"]
Oct 30 21:52:50.575617 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:50.575596    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-zskrq"
Oct 30 21:52:50.613087 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:50.613057    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-chgcq"]
Oct 30 21:52:50.613597 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:52:50.613570    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod147f4f98_ac6a_4b58_8750_a8410dd6324e.slice/crio-da02c9b461cfd4e81fb785512caf1f4ca215538b8615a90c6f288e516075fd27 WatchSource:0}: Error finding container da02c9b461cfd4e81fb785512caf1f4ca215538b8615a90c6f288e516075fd27: Status 404 returned error can't find the container with id da02c9b461cfd4e81fb785512caf1f4ca215538b8615a90c6f288e516075fd27
Oct 30 21:52:50.614885 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:50.614831    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-ddh7p"]
Oct 30 21:52:50.615741 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:50.615701    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:52:50.615821 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:50.615778    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:52:50.615877 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:50.615817    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:52:50.626764 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:50.626743    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-2d1f91-openshift-default-dockercfg-zn7vw\""
Oct 30 21:52:50.628146 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:52:50.628123    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba3aed70_33f3_4f0f_87cd_a8c19c347cd8.slice/crio-0f158d80b547a3963e47a1e8f8746f9889c64e9a4ed8d0d8c9e2d6b78d7fbc54 WatchSource:0}: Error finding container 0f158d80b547a3963e47a1e8f8746f9889c64e9a4ed8d0d8c9e2d6b78d7fbc54: Status 404 returned error can't find the container with id 0f158d80b547a3963e47a1e8f8746f9889c64e9a4ed8d0d8c9e2d6b78d7fbc54
Oct 30 21:52:50.630836 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:50.630795    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:52:50.630927 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:50.630910    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:52:50.630981 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:50.630951    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:52:50.638471 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:50.636335    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-zskrq"]
Oct 30 21:52:50.642557 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:50.638970    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-chgcq" event={"ID":"147f4f98-ac6a-4b58-8750-a8410dd6324e","Type":"ContainerStarted","Data":"da02c9b461cfd4e81fb785512caf1f4ca215538b8615a90c6f288e516075fd27"}
Oct 30 21:52:50.642557 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:50.640515    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-ddh7p" event={"ID":"ba3aed70-33f3-4f0f-87cd-a8c19c347cd8","Type":"ContainerStarted","Data":"0f158d80b547a3963e47a1e8f8746f9889c64e9a4ed8d0d8c9e2d6b78d7fbc54"}
Oct 30 21:52:50.671455 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:50.671425    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4z24f\" (UniqueName: \"kubernetes.io/projected/5eeacb87-9f34-4314-80f6-d538c5e843fa-kube-api-access-4z24f\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-zskrq\" (UID: \"5eeacb87-9f34-4314-80f6-d538c5e843fa\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-zskrq"
Oct 30 21:52:50.671604 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:50.671489    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5eeacb87-9f34-4314-80f6-d538c5e843fa-workload-certs\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-zskrq\" (UID: \"5eeacb87-9f34-4314-80f6-d538c5e843fa\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-zskrq"
Oct 30 21:52:50.671604 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:50.671557    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5eeacb87-9f34-4314-80f6-d538c5e843fa-istio-token\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-zskrq\" (UID: \"5eeacb87-9f34-4314-80f6-d538c5e843fa\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-zskrq"
Oct 30 21:52:50.671604 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:50.671588    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5eeacb87-9f34-4314-80f6-d538c5e843fa-istio-data\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-zskrq\" (UID: \"5eeacb87-9f34-4314-80f6-d538c5e843fa\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-zskrq"
Oct 30 21:52:50.671777 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:50.671659    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eeacb87-9f34-4314-80f6-d538c5e843fa-workload-socket\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-zskrq\" (UID: \"5eeacb87-9f34-4314-80f6-d538c5e843fa\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-zskrq"
Oct 30 21:52:50.671777 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:50.671691    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eeacb87-9f34-4314-80f6-d538c5e843fa-credential-socket\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-zskrq\" (UID: \"5eeacb87-9f34-4314-80f6-d538c5e843fa\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-zskrq"
Oct 30 21:52:50.671878 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:50.671777    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5eeacb87-9f34-4314-80f6-d538c5e843fa-istiod-ca-cert\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-zskrq\" (UID: \"5eeacb87-9f34-4314-80f6-d538c5e843fa\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-zskrq"
Oct 30 21:52:50.671878 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:50.671806    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5eeacb87-9f34-4314-80f6-d538c5e843fa-istio-podinfo\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-zskrq\" (UID: \"5eeacb87-9f34-4314-80f6-d538c5e843fa\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-zskrq"
Oct 30 21:52:50.671878 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:50.671854    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5eeacb87-9f34-4314-80f6-d538c5e843fa-istio-envoy\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-zskrq\" (UID: \"5eeacb87-9f34-4314-80f6-d538c5e843fa\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-zskrq"
Oct 30 21:52:50.773101 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:50.773013    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eeacb87-9f34-4314-80f6-d538c5e843fa-workload-socket\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-zskrq\" (UID: \"5eeacb87-9f34-4314-80f6-d538c5e843fa\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-zskrq"
Oct 30 21:52:50.773101 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:50.773064    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eeacb87-9f34-4314-80f6-d538c5e843fa-credential-socket\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-zskrq\" (UID: \"5eeacb87-9f34-4314-80f6-d538c5e843fa\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-zskrq"
Oct 30 21:52:50.775171 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:50.773504    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eeacb87-9f34-4314-80f6-d538c5e843fa-workload-socket\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-zskrq\" (UID: \"5eeacb87-9f34-4314-80f6-d538c5e843fa\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-zskrq"
Oct 30 21:52:50.775171 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:50.773540    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eeacb87-9f34-4314-80f6-d538c5e843fa-credential-socket\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-zskrq\" (UID: \"5eeacb87-9f34-4314-80f6-d538c5e843fa\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-zskrq"
Oct 30 21:52:50.775171 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:50.773851    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5eeacb87-9f34-4314-80f6-d538c5e843fa-istiod-ca-cert\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-zskrq\" (UID: \"5eeacb87-9f34-4314-80f6-d538c5e843fa\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-zskrq"
Oct 30 21:52:50.775171 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:50.773887    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5eeacb87-9f34-4314-80f6-d538c5e843fa-istio-podinfo\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-zskrq\" (UID: \"5eeacb87-9f34-4314-80f6-d538c5e843fa\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-zskrq"
Oct 30 21:52:50.775171 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:50.773932    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5eeacb87-9f34-4314-80f6-d538c5e843fa-istio-envoy\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-zskrq\" (UID: \"5eeacb87-9f34-4314-80f6-d538c5e843fa\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-zskrq"
Oct 30 21:52:50.775171 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:50.773989    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4z24f\" (UniqueName: \"kubernetes.io/projected/5eeacb87-9f34-4314-80f6-d538c5e843fa-kube-api-access-4z24f\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-zskrq\" (UID: \"5eeacb87-9f34-4314-80f6-d538c5e843fa\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-zskrq"
Oct 30 21:52:50.775171 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:50.774034    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5eeacb87-9f34-4314-80f6-d538c5e843fa-workload-certs\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-zskrq\" (UID: \"5eeacb87-9f34-4314-80f6-d538c5e843fa\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-zskrq"
Oct 30 21:52:50.775171 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:50.774078    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5eeacb87-9f34-4314-80f6-d538c5e843fa-istio-token\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-zskrq\" (UID: \"5eeacb87-9f34-4314-80f6-d538c5e843fa\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-zskrq"
Oct 30 21:52:50.775171 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:50.774110    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5eeacb87-9f34-4314-80f6-d538c5e843fa-istio-data\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-zskrq\" (UID: \"5eeacb87-9f34-4314-80f6-d538c5e843fa\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-zskrq"
Oct 30 21:52:50.775171 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:50.774456    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5eeacb87-9f34-4314-80f6-d538c5e843fa-istio-data\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-zskrq\" (UID: \"5eeacb87-9f34-4314-80f6-d538c5e843fa\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-zskrq"
Oct 30 21:52:50.775171 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:50.774622    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5eeacb87-9f34-4314-80f6-d538c5e843fa-istiod-ca-cert\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-zskrq\" (UID: \"5eeacb87-9f34-4314-80f6-d538c5e843fa\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-zskrq"
Oct 30 21:52:50.775171 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:50.775168    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5eeacb87-9f34-4314-80f6-d538c5e843fa-workload-certs\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-zskrq\" (UID: \"5eeacb87-9f34-4314-80f6-d538c5e843fa\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-zskrq"
Oct 30 21:52:50.777389 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:50.777349    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5eeacb87-9f34-4314-80f6-d538c5e843fa-istio-podinfo\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-zskrq\" (UID: \"5eeacb87-9f34-4314-80f6-d538c5e843fa\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-zskrq"
Oct 30 21:52:50.777677 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:50.777654    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5eeacb87-9f34-4314-80f6-d538c5e843fa-istio-envoy\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-zskrq\" (UID: \"5eeacb87-9f34-4314-80f6-d538c5e843fa\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-zskrq"
Oct 30 21:52:50.832108 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:50.832040    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5eeacb87-9f34-4314-80f6-d538c5e843fa-istio-token\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-zskrq\" (UID: \"5eeacb87-9f34-4314-80f6-d538c5e843fa\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-zskrq"
Oct 30 21:52:50.832108 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:50.832094    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4z24f\" (UniqueName: \"kubernetes.io/projected/5eeacb87-9f34-4314-80f6-d538c5e843fa-kube-api-access-4z24f\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-zskrq\" (UID: \"5eeacb87-9f34-4314-80f6-d538c5e843fa\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-zskrq"
Oct 30 21:52:50.889218 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:50.889172    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-zskrq"
Oct 30 21:52:51.380140 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:51.380105    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-zskrq"]
Oct 30 21:52:51.387190 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:52:51.387162    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5eeacb87_9f34_4314_80f6_d538c5e843fa.slice/crio-416eb4199ede4a69ede54e9ac404afe2e804413f90e9f7a5e47c499473a04741 WatchSource:0}: Error finding container 416eb4199ede4a69ede54e9ac404afe2e804413f90e9f7a5e47c499473a04741: Status 404 returned error can't find the container with id 416eb4199ede4a69ede54e9ac404afe2e804413f90e9f7a5e47c499473a04741
Oct 30 21:52:51.389447 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:51.389413    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:52:51.389549 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:51.389493    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:52:51.389549 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:51.389542    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:52:51.647677 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:51.647566    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-chgcq" event={"ID":"147f4f98-ac6a-4b58-8750-a8410dd6324e","Type":"ContainerStarted","Data":"0691492afd2aebc1e5aa04014c2805dfd67ff4e3087e09d2d5a432f20c862d05"}
Oct 30 21:52:51.649798 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:51.649773    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-ddh7p" event={"ID":"ba3aed70-33f3-4f0f-87cd-a8c19c347cd8","Type":"ContainerStarted","Data":"40776579af3656e34fb53d026a34a050339786ec5e6583ede8f3d72cf8844b91"}
Oct 30 21:52:51.651616 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:51.651570    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-zskrq" event={"ID":"5eeacb87-9f34-4314-80f6-d538c5e843fa","Type":"ContainerStarted","Data":"e57f49eb55bc61870711f23aabe0f47acf4b5e58c65d60588e93417e4590abdf"}
Oct 30 21:52:51.651728 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:51.651640    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-zskrq" event={"ID":"5eeacb87-9f34-4314-80f6-d538c5e843fa","Type":"ContainerStarted","Data":"416eb4199ede4a69ede54e9ac404afe2e804413f90e9f7a5e47c499473a04741"}
Oct 30 21:52:51.760856 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:51.760804    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-chgcq" podStartSLOduration=2.760790869 podStartE2EDuration="2.760790869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:52:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:52:51.759388575 +0000 UTC m=+625.372158332" watchObservedRunningTime="2026-10-30 21:52:51.760790869 +0000 UTC m=+625.373560626"
Oct 30 21:52:51.875987 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:51.875938    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-ddh7p" podStartSLOduration=2.875925062 podStartE2EDuration="2.875925062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:52:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:52:51.874320356 +0000 UTC m=+625.487090114" watchObservedRunningTime="2026-10-30 21:52:51.875925062 +0000 UTC m=+625.488694820"
Oct 30 21:52:51.889918 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:51.889891    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-zskrq"
Oct 30 21:52:51.986465 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:51.986387    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-zskrq" podStartSLOduration=1.986375696 podStartE2EDuration="1.986375696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:52:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:52:51.985381168 +0000 UTC m=+625.598150945" watchObservedRunningTime="2026-10-30 21:52:51.986375696 +0000 UTC m=+625.599145454"
Oct 30 21:52:52.129873 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:52.129838    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-chgcq"
Oct 30 21:52:52.135847 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:52.135821    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-chgcq"
Oct 30 21:52:52.215559 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:52.215499    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-ddh7p"
Oct 30 21:52:52.220719 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:52.220695    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-ddh7p"
Oct 30 21:52:52.657072 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:52.657031    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-ddh7p"
Oct 30 21:52:52.657545 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:52.657082    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-chgcq"
Oct 30 21:52:52.657753 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:52.657735    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-chgcq"
Oct 30 21:52:52.657811 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:52.657783    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-ddh7p"
Oct 30 21:52:52.896696 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:52.896665    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-zskrq"
Oct 30 21:52:53.663105 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:53.663075    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-zskrq"
Oct 30 21:52:53.663880 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:53.663865    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-zskrq"
Oct 30 21:52:54.665357 ip-10-0-130-60 kubenswrapper[5061]: E1030 21:52:54.665319    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 21:52:58.657261 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:58.657230    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-14af5c-85d68c5546-r4slw"]
Oct 30 21:52:58.661443 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:58.661423    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-14af5c-85d68c5546-r4slw"
Oct 30 21:52:58.758034 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:58.758005    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhjc6\" (UniqueName: \"kubernetes.io/projected/6977b446-b5f9-4f27-8d41-a4942379c9b2-kube-api-access-fhjc6\") pod \"payload-processing-e2e-worker-w6-14af5c-85d68c5546-r4slw\" (UID: \"6977b446-b5f9-4f27-8d41-a4942379c9b2\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-14af5c-85d68c5546-r4slw"
Oct 30 21:52:58.758183 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:58.758075    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6977b446-b5f9-4f27-8d41-a4942379c9b2-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-14af5c-85d68c5546-r4slw\" (UID: \"6977b446-b5f9-4f27-8d41-a4942379c9b2\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-14af5c-85d68c5546-r4slw"
Oct 30 21:52:58.758495 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:58.758475    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-14af5c-dockercfg-q5p8b\""
Oct 30 21:52:58.758724 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:58.758706    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-14af5c\""
Oct 30 21:52:58.760842 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:58.760824    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-14af5c-85d68c5546-r4slw"]
Oct 30 21:52:58.858695 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:58.858665    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhjc6\" (UniqueName: \"kubernetes.io/projected/6977b446-b5f9-4f27-8d41-a4942379c9b2-kube-api-access-fhjc6\") pod \"payload-processing-e2e-worker-w6-14af5c-85d68c5546-r4slw\" (UID: \"6977b446-b5f9-4f27-8d41-a4942379c9b2\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-14af5c-85d68c5546-r4slw"
Oct 30 21:52:58.858859 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:58.858748    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6977b446-b5f9-4f27-8d41-a4942379c9b2-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-14af5c-85d68c5546-r4slw\" (UID: \"6977b446-b5f9-4f27-8d41-a4942379c9b2\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-14af5c-85d68c5546-r4slw"
Oct 30 21:52:58.859487 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:58.859469    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6977b446-b5f9-4f27-8d41-a4942379c9b2-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-14af5c-85d68c5546-r4slw\" (UID: \"6977b446-b5f9-4f27-8d41-a4942379c9b2\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-14af5c-85d68c5546-r4slw"
Oct 30 21:52:58.919003 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:58.918937    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhjc6\" (UniqueName: \"kubernetes.io/projected/6977b446-b5f9-4f27-8d41-a4942379c9b2-kube-api-access-fhjc6\") pod \"payload-processing-e2e-worker-w6-14af5c-85d68c5546-r4slw\" (UID: \"6977b446-b5f9-4f27-8d41-a4942379c9b2\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-14af5c-85d68c5546-r4slw"
Oct 30 21:52:58.975670 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:58.975640    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-14af5c-85d68c5546-r4slw"
Oct 30 21:52:59.348990 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:59.348962    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-14af5c-85d68c5546-r4slw"]
Oct 30 21:52:59.354123 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:52:59.354096    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6977b446_b5f9_4f27_8d41_a4942379c9b2.slice/crio-7ae05b3119e8374fef5f9e038d0727a741f7d0d1b29a2826c0d0ebf5662fa48c WatchSource:0}: Error finding container 7ae05b3119e8374fef5f9e038d0727a741f7d0d1b29a2826c0d0ebf5662fa48c: Status 404 returned error can't find the container with id 7ae05b3119e8374fef5f9e038d0727a741f7d0d1b29a2826c0d0ebf5662fa48c
Oct 30 21:52:59.693859 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:59.693786    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-14af5c-85d68c5546-r4slw" event={"ID":"6977b446-b5f9-4f27-8d41-a4942379c9b2","Type":"ContainerStarted","Data":"86d8a666c1779dda82bfa30c16d519a8d51af7e4b1517f6f22318facc31637d1"}
Oct 30 21:52:59.693859 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:59.693835    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-14af5c-85d68c5546-r4slw" event={"ID":"6977b446-b5f9-4f27-8d41-a4942379c9b2","Type":"ContainerStarted","Data":"7ae05b3119e8374fef5f9e038d0727a741f7d0d1b29a2826c0d0ebf5662fa48c"}
Oct 30 21:52:59.694262 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:59.693920    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-14af5c-85d68c5546-r4slw"
Oct 30 21:52:59.810012 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:52:59.809963    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w6-14af5c-85d68c5546-r4slw" podStartSLOduration=1.8099506330000001 podStartE2EDuration="1.809950633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:52:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:52:59.808606926 +0000 UTC m=+633.421376684" watchObservedRunningTime="2026-10-30 21:52:59.809950633 +0000 UTC m=+633.422720391"
Oct 30 21:53:03.108500 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:03.108464    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-5402dd-86b6d5dd95-xzj89"]
Oct 30 21:53:03.112727 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:03.112706    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-5402dd-86b6d5dd95-xzj89"
Oct 30 21:53:03.220832 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:03.204140    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wxgf\" (UniqueName: \"kubernetes.io/projected/61e905c3-97ef-4f17-aba7-0ea699720853-kube-api-access-4wxgf\") pod \"payload-pre-processing-e2e-worker-w5-5402dd-86b6d5dd95-xzj89\" (UID: \"61e905c3-97ef-4f17-aba7-0ea699720853\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-5402dd-86b6d5dd95-xzj89"
Oct 30 21:53:03.221370 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:03.221329    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61e905c3-97ef-4f17-aba7-0ea699720853-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-5402dd-86b6d5dd95-xzj89\" (UID: \"61e905c3-97ef-4f17-aba7-0ea699720853\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-5402dd-86b6d5dd95-xzj89"
Oct 30 21:53:03.221627 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:03.214293    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-5402dd-86b6d5dd95-xzj89"]
Oct 30 21:53:03.221788 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:03.216423    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-5402dd-dockercfg-c9p5b\""
Oct 30 21:53:03.221883 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:03.211706    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-5402dd\""
Oct 30 21:53:03.276654 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:03.276625    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-5402dd-57dc966bb-4x6rt"]
Oct 30 21:53:03.280952 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:03.280932    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-5402dd-57dc966bb-4x6rt"
Oct 30 21:53:03.323017 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:03.322993    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wxgf\" (UniqueName: \"kubernetes.io/projected/61e905c3-97ef-4f17-aba7-0ea699720853-kube-api-access-4wxgf\") pod \"payload-pre-processing-e2e-worker-w5-5402dd-86b6d5dd95-xzj89\" (UID: \"61e905c3-97ef-4f17-aba7-0ea699720853\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-5402dd-86b6d5dd95-xzj89"
Oct 30 21:53:03.323157 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:03.323141    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61e905c3-97ef-4f17-aba7-0ea699720853-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-5402dd-86b6d5dd95-xzj89\" (UID: \"61e905c3-97ef-4f17-aba7-0ea699720853\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-5402dd-86b6d5dd95-xzj89"
Oct 30 21:53:03.323835 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:03.323818    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61e905c3-97ef-4f17-aba7-0ea699720853-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-5402dd-86b6d5dd95-xzj89\" (UID: \"61e905c3-97ef-4f17-aba7-0ea699720853\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-5402dd-86b6d5dd95-xzj89"
Oct 30 21:53:03.337222 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:03.337194    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-5402dd-57dc966bb-4x6rt"]
Oct 30 21:53:03.379299 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:03.379240    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wxgf\" (UniqueName: \"kubernetes.io/projected/61e905c3-97ef-4f17-aba7-0ea699720853-kube-api-access-4wxgf\") pod \"payload-pre-processing-e2e-worker-w5-5402dd-86b6d5dd95-xzj89\" (UID: \"61e905c3-97ef-4f17-aba7-0ea699720853\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-5402dd-86b6d5dd95-xzj89"
Oct 30 21:53:03.424224 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:03.424194    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7z59\" (UniqueName: \"kubernetes.io/projected/a02d777c-062a-4182-81e7-1952fa5924f5-kube-api-access-l7z59\") pod \"payload-processing-e2e-worker-w5-5402dd-57dc966bb-4x6rt\" (UID: \"a02d777c-062a-4182-81e7-1952fa5924f5\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-5402dd-57dc966bb-4x6rt"
Oct 30 21:53:03.424344 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:03.424327    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a02d777c-062a-4182-81e7-1952fa5924f5-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-5402dd-57dc966bb-4x6rt\" (UID: \"a02d777c-062a-4182-81e7-1952fa5924f5\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-5402dd-57dc966bb-4x6rt"
Oct 30 21:53:03.425967 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:03.425949    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-5402dd-86b6d5dd95-xzj89"
Oct 30 21:53:03.525932 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:03.525896    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7z59\" (UniqueName: \"kubernetes.io/projected/a02d777c-062a-4182-81e7-1952fa5924f5-kube-api-access-l7z59\") pod \"payload-processing-e2e-worker-w5-5402dd-57dc966bb-4x6rt\" (UID: \"a02d777c-062a-4182-81e7-1952fa5924f5\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-5402dd-57dc966bb-4x6rt"
Oct 30 21:53:03.526065 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:03.526020    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a02d777c-062a-4182-81e7-1952fa5924f5-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-5402dd-57dc966bb-4x6rt\" (UID: \"a02d777c-062a-4182-81e7-1952fa5924f5\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-5402dd-57dc966bb-4x6rt"
Oct 30 21:53:03.526971 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:03.526941    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a02d777c-062a-4182-81e7-1952fa5924f5-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-5402dd-57dc966bb-4x6rt\" (UID: \"a02d777c-062a-4182-81e7-1952fa5924f5\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-5402dd-57dc966bb-4x6rt"
Oct 30 21:53:03.582044 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:03.582019    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7z59\" (UniqueName: \"kubernetes.io/projected/a02d777c-062a-4182-81e7-1952fa5924f5-kube-api-access-l7z59\") pod \"payload-processing-e2e-worker-w5-5402dd-57dc966bb-4x6rt\" (UID: \"a02d777c-062a-4182-81e7-1952fa5924f5\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-5402dd-57dc966bb-4x6rt"
Oct 30 21:53:03.595430 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:03.595391    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-5402dd-57dc966bb-4x6rt"
Oct 30 21:53:03.792442 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:03.792411    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-5402dd-86b6d5dd95-xzj89"]
Oct 30 21:53:03.797980 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:53:03.797952    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61e905c3_97ef_4f17_aba7_0ea699720853.slice/crio-0c03759fb1f54414154446c8b6be91977853e0da3fa67d01b0a0381e1f18b691 WatchSource:0}: Error finding container 0c03759fb1f54414154446c8b6be91977853e0da3fa67d01b0a0381e1f18b691: Status 404 returned error can't find the container with id 0c03759fb1f54414154446c8b6be91977853e0da3fa67d01b0a0381e1f18b691
Oct 30 21:53:03.967296 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:03.967225    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-5402dd-57dc966bb-4x6rt"]
Oct 30 21:53:03.970328 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:53:03.970296    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda02d777c_062a_4182_81e7_1952fa5924f5.slice/crio-bc89d81f3bee8966fa5ea28d4bf0a6b8b712ac5f594357c9997107b84c790dc5 WatchSource:0}: Error finding container bc89d81f3bee8966fa5ea28d4bf0a6b8b712ac5f594357c9997107b84c790dc5: Status 404 returned error can't find the container with id bc89d81f3bee8966fa5ea28d4bf0a6b8b712ac5f594357c9997107b84c790dc5
Oct 30 21:53:04.726516 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:04.726473    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-5402dd-57dc966bb-4x6rt" event={"ID":"a02d777c-062a-4182-81e7-1952fa5924f5","Type":"ContainerStarted","Data":"5c6bc908e527fe0e7d23233fa6a218eeddb3b58cc9984d5095231dcdee2dc100"}
Oct 30 21:53:04.727126 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:04.726584    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-5402dd-57dc966bb-4x6rt" event={"ID":"a02d777c-062a-4182-81e7-1952fa5924f5","Type":"ContainerStarted","Data":"bc89d81f3bee8966fa5ea28d4bf0a6b8b712ac5f594357c9997107b84c790dc5"}
Oct 30 21:53:04.727126 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:04.726618    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-5402dd-57dc966bb-4x6rt"
Oct 30 21:53:04.733797 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:04.733758    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-5402dd-86b6d5dd95-xzj89" event={"ID":"61e905c3-97ef-4f17-aba7-0ea699720853","Type":"ContainerStarted","Data":"e8e3bf7f3a42e9c86d0d52fafd396750222aa6219defa39e2431640714f71120"}
Oct 30 21:53:04.733980 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:04.733824    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-5402dd-86b6d5dd95-xzj89" event={"ID":"61e905c3-97ef-4f17-aba7-0ea699720853","Type":"ContainerStarted","Data":"0c03759fb1f54414154446c8b6be91977853e0da3fa67d01b0a0381e1f18b691"}
Oct 30 21:53:04.734047 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:04.734031    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-5402dd-86b6d5dd95-xzj89"
Oct 30 21:53:04.840867 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:04.840819    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w5-5402dd-57dc966bb-4x6rt" podStartSLOduration=1.840801658 podStartE2EDuration="1.840801658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:53:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:53:04.838434727 +0000 UTC m=+638.451204484" watchObservedRunningTime="2026-10-30 21:53:04.840801658 +0000 UTC m=+638.453571415"
Oct 30 21:53:04.948942 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:04.948890    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-5402dd-86b6d5dd95-xzj89" podStartSLOduration=1.9488772989999998 podStartE2EDuration="1.948877299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:53:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:53:04.94743665 +0000 UTC m=+638.560206409" watchObservedRunningTime="2026-10-30 21:53:04.948877299 +0000 UTC m=+638.561647053"
Oct 30 21:53:08.002476 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:08.002428    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-f776cb-74d8d8c899-hnzj8"]
Oct 30 21:53:08.021251 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:08.019027    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f776cb-74d8d8c899-hnzj8"
Oct 30 21:53:08.067388 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:08.067356    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-f776cb-74d8d8c899-hnzj8"]
Oct 30 21:53:08.075880 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:08.075241    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8hc9\" (UniqueName: \"kubernetes.io/projected/fd36ba7d-191a-4b1d-bbee-7d34510ecdc0-kube-api-access-j8hc9\") pod \"payload-pre-processing-e2e-worker-w3-f776cb-74d8d8c899-hnzj8\" (UID: \"fd36ba7d-191a-4b1d-bbee-7d34510ecdc0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f776cb-74d8d8c899-hnzj8"
Oct 30 21:53:08.075880 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:08.075305    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd36ba7d-191a-4b1d-bbee-7d34510ecdc0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-f776cb-74d8d8c899-hnzj8\" (UID: \"fd36ba7d-191a-4b1d-bbee-7d34510ecdc0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f776cb-74d8d8c899-hnzj8"
Oct 30 21:53:08.111746 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:08.111721    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-f776cb\""
Oct 30 21:53:08.120222 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:08.120188    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-f776cb-dockercfg-vf8v4\""
Oct 30 21:53:08.176721 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:08.176689    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8hc9\" (UniqueName: \"kubernetes.io/projected/fd36ba7d-191a-4b1d-bbee-7d34510ecdc0-kube-api-access-j8hc9\") pod \"payload-pre-processing-e2e-worker-w3-f776cb-74d8d8c899-hnzj8\" (UID: \"fd36ba7d-191a-4b1d-bbee-7d34510ecdc0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f776cb-74d8d8c899-hnzj8"
Oct 30 21:53:08.176909 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:08.176735    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd36ba7d-191a-4b1d-bbee-7d34510ecdc0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-f776cb-74d8d8c899-hnzj8\" (UID: \"fd36ba7d-191a-4b1d-bbee-7d34510ecdc0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f776cb-74d8d8c899-hnzj8"
Oct 30 21:53:08.177496 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:08.177470    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd36ba7d-191a-4b1d-bbee-7d34510ecdc0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-f776cb-74d8d8c899-hnzj8\" (UID: \"fd36ba7d-191a-4b1d-bbee-7d34510ecdc0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f776cb-74d8d8c899-hnzj8"
Oct 30 21:53:08.233772 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:08.233736    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8hc9\" (UniqueName: \"kubernetes.io/projected/fd36ba7d-191a-4b1d-bbee-7d34510ecdc0-kube-api-access-j8hc9\") pod \"payload-pre-processing-e2e-worker-w3-f776cb-74d8d8c899-hnzj8\" (UID: \"fd36ba7d-191a-4b1d-bbee-7d34510ecdc0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f776cb-74d8d8c899-hnzj8"
Oct 30 21:53:08.343092 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:08.343056    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f776cb-74d8d8c899-hnzj8"
Oct 30 21:53:08.732853 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:08.732775    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-f776cb-74d8d8c899-hnzj8"]
Oct 30 21:53:08.737926 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:53:08.737888    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd36ba7d_191a_4b1d_bbee_7d34510ecdc0.slice/crio-d644bea2ed2074747c211130c5dd102eac419ba934bba8fb4636b16ff08f9ba8 WatchSource:0}: Error finding container d644bea2ed2074747c211130c5dd102eac419ba934bba8fb4636b16ff08f9ba8: Status 404 returned error can't find the container with id d644bea2ed2074747c211130c5dd102eac419ba934bba8fb4636b16ff08f9ba8
Oct 30 21:53:08.757824 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:08.757799    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f776cb-74d8d8c899-hnzj8" event={"ID":"fd36ba7d-191a-4b1d-bbee-7d34510ecdc0","Type":"ContainerStarted","Data":"d644bea2ed2074747c211130c5dd102eac419ba934bba8fb4636b16ff08f9ba8"}
Oct 30 21:53:09.764548 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:09.764496    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f776cb-74d8d8c899-hnzj8" event={"ID":"fd36ba7d-191a-4b1d-bbee-7d34510ecdc0","Type":"ContainerStarted","Data":"ca6df9a8af50dd6294921abac5b1e94220fbec69132e2cb7df6549cb2b91afc4"}
Oct 30 21:53:09.765057 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:09.764684    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f776cb-74d8d8c899-hnzj8"
Oct 30 21:53:09.873426 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:09.873376    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f776cb-74d8d8c899-hnzj8" podStartSLOduration=2.8733640449999998 podStartE2EDuration="2.873364045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:53:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:53:09.872106439 +0000 UTC m=+643.484876197" watchObservedRunningTime="2026-10-30 21:53:09.873364045 +0000 UTC m=+643.486133802"
Oct 30 21:53:10.544223 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:10.544187    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-2d1f91-df6cc69d8-wzwll"]
Oct 30 21:53:10.548577 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:10.548553    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-2d1f91-df6cc69d8-wzwll"
Oct 30 21:53:10.598226 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:10.598196    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-2d1f91\""
Oct 30 21:53:10.598473 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:10.598450    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-2d1f91-dockercfg-vpslr\""
Oct 30 21:53:10.606582 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:10.606560    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-2d1f91-df6cc69d8-wzwll"]
Oct 30 21:53:10.612660 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:10.612634    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxpf4\" (UniqueName: \"kubernetes.io/projected/d71ce3d8-00ba-4ff3-a870-6dd143dbaf93-kube-api-access-zxpf4\") pod \"payload-processing-e2e-worker-w0-2d1f91-df6cc69d8-wzwll\" (UID: \"d71ce3d8-00ba-4ff3-a870-6dd143dbaf93\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-2d1f91-df6cc69d8-wzwll"
Oct 30 21:53:10.614706 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:10.612775    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d71ce3d8-00ba-4ff3-a870-6dd143dbaf93-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-2d1f91-df6cc69d8-wzwll\" (UID: \"d71ce3d8-00ba-4ff3-a870-6dd143dbaf93\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-2d1f91-df6cc69d8-wzwll"
Oct 30 21:53:10.701581 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:10.701553    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-14af5c-85d68c5546-r4slw"
Oct 30 21:53:10.713393 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:10.713365    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxpf4\" (UniqueName: \"kubernetes.io/projected/d71ce3d8-00ba-4ff3-a870-6dd143dbaf93-kube-api-access-zxpf4\") pod \"payload-processing-e2e-worker-w0-2d1f91-df6cc69d8-wzwll\" (UID: \"d71ce3d8-00ba-4ff3-a870-6dd143dbaf93\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-2d1f91-df6cc69d8-wzwll"
Oct 30 21:53:10.713567 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:10.713474    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d71ce3d8-00ba-4ff3-a870-6dd143dbaf93-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-2d1f91-df6cc69d8-wzwll\" (UID: \"d71ce3d8-00ba-4ff3-a870-6dd143dbaf93\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-2d1f91-df6cc69d8-wzwll"
Oct 30 21:53:10.714217 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:10.714196    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d71ce3d8-00ba-4ff3-a870-6dd143dbaf93-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-2d1f91-df6cc69d8-wzwll\" (UID: \"d71ce3d8-00ba-4ff3-a870-6dd143dbaf93\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-2d1f91-df6cc69d8-wzwll"
Oct 30 21:53:10.767779 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:10.767752    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxpf4\" (UniqueName: \"kubernetes.io/projected/d71ce3d8-00ba-4ff3-a870-6dd143dbaf93-kube-api-access-zxpf4\") pod \"payload-processing-e2e-worker-w0-2d1f91-df6cc69d8-wzwll\" (UID: \"d71ce3d8-00ba-4ff3-a870-6dd143dbaf93\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-2d1f91-df6cc69d8-wzwll"
Oct 30 21:53:10.863424 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:10.863347    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-2d1f91-df6cc69d8-wzwll"
Oct 30 21:53:11.235059 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:11.234980    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-2d1f91-df6cc69d8-wzwll"]
Oct 30 21:53:11.239458 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:53:11.239432    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd71ce3d8_00ba_4ff3_a870_6dd143dbaf93.slice/crio-fc1dc6f7f21aa0ae9b1c8c2fd38d539ee8c72447d6c722dea7e193aeddf4367d WatchSource:0}: Error finding container fc1dc6f7f21aa0ae9b1c8c2fd38d539ee8c72447d6c722dea7e193aeddf4367d: Status 404 returned error can't find the container with id fc1dc6f7f21aa0ae9b1c8c2fd38d539ee8c72447d6c722dea7e193aeddf4367d
Oct 30 21:53:11.776851 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:11.776813    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-2d1f91-df6cc69d8-wzwll" event={"ID":"d71ce3d8-00ba-4ff3-a870-6dd143dbaf93","Type":"ContainerStarted","Data":"41f367b9e84d1295d677672d630e913cd2bff627bee86bd938a7d31706bc6628"}
Oct 30 21:53:11.777267 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:11.776972    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-2d1f91-df6cc69d8-wzwll"
Oct 30 21:53:11.777267 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:11.777014    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-2d1f91-df6cc69d8-wzwll" event={"ID":"d71ce3d8-00ba-4ff3-a870-6dd143dbaf93","Type":"ContainerStarted","Data":"fc1dc6f7f21aa0ae9b1c8c2fd38d539ee8c72447d6c722dea7e193aeddf4367d"}
Oct 30 21:53:11.890740 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:11.890687    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-2d1f91-df6cc69d8-wzwll" podStartSLOduration=1.890673932 podStartE2EDuration="1.890673932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:53:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:53:11.888710581 +0000 UTC m=+645.501480352" watchObservedRunningTime="2026-10-30 21:53:11.890673932 +0000 UTC m=+645.503443690"
Oct 30 21:53:15.742806 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:15.742777    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-5402dd-86b6d5dd95-xzj89"
Oct 30 21:53:15.743208 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:15.742873    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-5402dd-57dc966bb-4x6rt"
Oct 30 21:53:17.321960 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:17.321897    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-5be0d1-6989644d59-xk2lk"]
Oct 30 21:53:17.326602 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:17.326580    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-5be0d1-6989644d59-xk2lk"
Oct 30 21:53:17.377577 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:17.377553    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-5be0d1\""
Oct 30 21:53:17.383499 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:17.383467    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kbrd\" (UniqueName: \"kubernetes.io/projected/36373ff4-1396-4b98-8f21-8e261b2e0268-kube-api-access-7kbrd\") pod \"payload-pre-processing-e2e-worker-w2-5be0d1-6989644d59-xk2lk\" (UID: \"36373ff4-1396-4b98-8f21-8e261b2e0268\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-5be0d1-6989644d59-xk2lk"
Oct 30 21:53:17.383666 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:17.383646    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36373ff4-1396-4b98-8f21-8e261b2e0268-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-5be0d1-6989644d59-xk2lk\" (UID: \"36373ff4-1396-4b98-8f21-8e261b2e0268\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-5be0d1-6989644d59-xk2lk"
Oct 30 21:53:17.424591 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:17.424516    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-5be0d1-dockercfg-492wc\""
Oct 30 21:53:17.427377 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:17.427341    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-5be0d1-6989644d59-xk2lk"]
Oct 30 21:53:17.502199 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:17.502157    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kbrd\" (UniqueName: \"kubernetes.io/projected/36373ff4-1396-4b98-8f21-8e261b2e0268-kube-api-access-7kbrd\") pod \"payload-pre-processing-e2e-worker-w2-5be0d1-6989644d59-xk2lk\" (UID: \"36373ff4-1396-4b98-8f21-8e261b2e0268\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-5be0d1-6989644d59-xk2lk"
Oct 30 21:53:17.503930 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:17.503901    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36373ff4-1396-4b98-8f21-8e261b2e0268-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-5be0d1-6989644d59-xk2lk\" (UID: \"36373ff4-1396-4b98-8f21-8e261b2e0268\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-5be0d1-6989644d59-xk2lk"
Oct 30 21:53:17.510134 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:17.504969    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36373ff4-1396-4b98-8f21-8e261b2e0268-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-5be0d1-6989644d59-xk2lk\" (UID: \"36373ff4-1396-4b98-8f21-8e261b2e0268\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-5be0d1-6989644d59-xk2lk"
Oct 30 21:53:17.562161 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:17.562124    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kbrd\" (UniqueName: \"kubernetes.io/projected/36373ff4-1396-4b98-8f21-8e261b2e0268-kube-api-access-7kbrd\") pod \"payload-pre-processing-e2e-worker-w2-5be0d1-6989644d59-xk2lk\" (UID: \"36373ff4-1396-4b98-8f21-8e261b2e0268\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-5be0d1-6989644d59-xk2lk"
Oct 30 21:53:17.642889 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:17.640087    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-5be0d1-6989644d59-xk2lk"
Oct 30 21:53:18.031319 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:18.031216    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-5be0d1-6989644d59-xk2lk"]
Oct 30 21:53:18.837557 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:18.837505    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-5be0d1-6989644d59-xk2lk" event={"ID":"36373ff4-1396-4b98-8f21-8e261b2e0268","Type":"ContainerStarted","Data":"415debea16a93bcde53c7d946768352b6cd0f0878606f3e6a4a7c1f530099f2f"}
Oct 30 21:53:18.837984 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:18.837571    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-5be0d1-6989644d59-xk2lk" event={"ID":"36373ff4-1396-4b98-8f21-8e261b2e0268","Type":"ContainerStarted","Data":"da56f7661420ce624855478867144dd6762771bc69911d3dcafc1e2b4c367241"}
Oct 30 21:53:18.837984 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:18.837654    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-5be0d1-6989644d59-xk2lk"
Oct 30 21:53:18.951308 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:18.951254    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-5be0d1-6989644d59-xk2lk" podStartSLOduration=1.9512414580000002 podStartE2EDuration="1.951241458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:53:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:53:18.949413238 +0000 UTC m=+652.562182996" watchObservedRunningTime="2026-10-30 21:53:18.951241458 +0000 UTC m=+652.564011216"
Oct 30 21:53:20.773744 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:20.773712    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f776cb-74d8d8c899-hnzj8"
Oct 30 21:53:22.785368 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:22.785339    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-2d1f91-df6cc69d8-wzwll"
Oct 30 21:53:24.416629 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:24.416591    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-14af5c-bd596f55c-f4z8t"]
Oct 30 21:53:24.422097 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:24.422065    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-14af5c-bd596f55c-f4z8t"
Oct 30 21:53:24.475946 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:24.475914    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w6-14af5c-serving-cert\""
Oct 30 21:53:24.476222 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:24.476200    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 30 21:53:24.498944 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:24.498910    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qc8g2\" (UniqueName: \"kubernetes.io/projected/4646779e-fc85-4860-adcf-9855d2d26fb1-kube-api-access-qc8g2\") pod \"maas-api-e2e-worker-w6-14af5c-bd596f55c-f4z8t\" (UID: \"4646779e-fc85-4860-adcf-9855d2d26fb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-14af5c-bd596f55c-f4z8t"
Oct 30 21:53:24.499095 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:24.499074    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4646779e-fc85-4860-adcf-9855d2d26fb1-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-14af5c-bd596f55c-f4z8t\" (UID: \"4646779e-fc85-4860-adcf-9855d2d26fb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-14af5c-bd596f55c-f4z8t"
Oct 30 21:53:24.499139 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:24.499127    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4646779e-fc85-4860-adcf-9855d2d26fb1-metrics-certs\") pod \"maas-api-e2e-worker-w6-14af5c-bd596f55c-f4z8t\" (UID: \"4646779e-fc85-4860-adcf-9855d2d26fb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-14af5c-bd596f55c-f4z8t"
Oct 30 21:53:24.499347 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:24.499327    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4646779e-fc85-4860-adcf-9855d2d26fb1-maas-api-tls\") pod \"maas-api-e2e-worker-w6-14af5c-bd596f55c-f4z8t\" (UID: \"4646779e-fc85-4860-adcf-9855d2d26fb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-14af5c-bd596f55c-f4z8t"
Oct 30 21:53:24.523834 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:24.523813    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:53:24.524075 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:24.524053    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-rsvrw\""
Oct 30 21:53:24.531021 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:24.530997    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-14af5c-bd596f55c-f4z8t"]
Oct 30 21:53:24.577776 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:24.577747    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:53:24.600922 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:24.600894    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4646779e-fc85-4860-adcf-9855d2d26fb1-maas-api-tls\") pod \"maas-api-e2e-worker-w6-14af5c-bd596f55c-f4z8t\" (UID: \"4646779e-fc85-4860-adcf-9855d2d26fb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-14af5c-bd596f55c-f4z8t"
Oct 30 21:53:24.601105 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:24.600938    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qc8g2\" (UniqueName: \"kubernetes.io/projected/4646779e-fc85-4860-adcf-9855d2d26fb1-kube-api-access-qc8g2\") pod \"maas-api-e2e-worker-w6-14af5c-bd596f55c-f4z8t\" (UID: \"4646779e-fc85-4860-adcf-9855d2d26fb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-14af5c-bd596f55c-f4z8t"
Oct 30 21:53:24.601105 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:24.601025    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4646779e-fc85-4860-adcf-9855d2d26fb1-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-14af5c-bd596f55c-f4z8t\" (UID: \"4646779e-fc85-4860-adcf-9855d2d26fb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-14af5c-bd596f55c-f4z8t"
Oct 30 21:53:24.601105 ip-10-0-130-60 kubenswrapper[5061]: E1030 21:53:24.601046    5061 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w6-14af5c-serving-cert: secret "maas-api-e2e-worker-w6-14af5c-serving-cert" not found
Oct 30 21:53:24.601277 ip-10-0-130-60 kubenswrapper[5061]: E1030 21:53:24.601116    5061 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4646779e-fc85-4860-adcf-9855d2d26fb1-maas-api-tls podName:4646779e-fc85-4860-adcf-9855d2d26fb1 nodeName:}" failed. No retries permitted until 2026-10-30 21:53:25.101095908 +0000 UTC m=+658.713865648 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4646779e-fc85-4860-adcf-9855d2d26fb1-maas-api-tls") pod "maas-api-e2e-worker-w6-14af5c-bd596f55c-f4z8t" (UID: "4646779e-fc85-4860-adcf-9855d2d26fb1") : secret "maas-api-e2e-worker-w6-14af5c-serving-cert" not found
Oct 30 21:53:24.601277 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:24.601204    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4646779e-fc85-4860-adcf-9855d2d26fb1-metrics-certs\") pod \"maas-api-e2e-worker-w6-14af5c-bd596f55c-f4z8t\" (UID: \"4646779e-fc85-4860-adcf-9855d2d26fb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-14af5c-bd596f55c-f4z8t"
Oct 30 21:53:24.601774 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:24.601756    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4646779e-fc85-4860-adcf-9855d2d26fb1-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-14af5c-bd596f55c-f4z8t\" (UID: \"4646779e-fc85-4860-adcf-9855d2d26fb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-14af5c-bd596f55c-f4z8t"
Oct 30 21:53:24.603705 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:24.603682    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4646779e-fc85-4860-adcf-9855d2d26fb1-metrics-certs\") pod \"maas-api-e2e-worker-w6-14af5c-bd596f55c-f4z8t\" (UID: \"4646779e-fc85-4860-adcf-9855d2d26fb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-14af5c-bd596f55c-f4z8t"
Oct 30 21:53:24.659440 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:24.659412    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qc8g2\" (UniqueName: \"kubernetes.io/projected/4646779e-fc85-4860-adcf-9855d2d26fb1-kube-api-access-qc8g2\") pod \"maas-api-e2e-worker-w6-14af5c-bd596f55c-f4z8t\" (UID: \"4646779e-fc85-4860-adcf-9855d2d26fb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-14af5c-bd596f55c-f4z8t"
Oct 30 21:53:24.679067 ip-10-0-130-60 kubenswrapper[5061]: E1030 21:53:24.678978    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 21:53:25.108047 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:25.108013    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4646779e-fc85-4860-adcf-9855d2d26fb1-maas-api-tls\") pod \"maas-api-e2e-worker-w6-14af5c-bd596f55c-f4z8t\" (UID: \"4646779e-fc85-4860-adcf-9855d2d26fb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-14af5c-bd596f55c-f4z8t"
Oct 30 21:53:25.110663 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:25.110635    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4646779e-fc85-4860-adcf-9855d2d26fb1-maas-api-tls\") pod \"maas-api-e2e-worker-w6-14af5c-bd596f55c-f4z8t\" (UID: \"4646779e-fc85-4860-adcf-9855d2d26fb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-14af5c-bd596f55c-f4z8t"
Oct 30 21:53:25.335935 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:25.335898    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-14af5c-bd596f55c-f4z8t"
Oct 30 21:53:25.730755 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:25.730650    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-14af5c-bd596f55c-f4z8t"]
Oct 30 21:53:25.737071 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:53:25.737041    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4646779e_fc85_4860_adcf_9855d2d26fb1.slice/crio-e2bfe1093bf39b6079c816f77109a76c143e595328f461ea11efdb3ca78d71db WatchSource:0}: Error finding container e2bfe1093bf39b6079c816f77109a76c143e595328f461ea11efdb3ca78d71db: Status 404 returned error can't find the container with id e2bfe1093bf39b6079c816f77109a76c143e595328f461ea11efdb3ca78d71db
Oct 30 21:53:25.738582 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:25.738563    5061 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 21:53:25.875461 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:25.875420    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-14af5c-bd596f55c-f4z8t" event={"ID":"4646779e-fc85-4860-adcf-9855d2d26fb1","Type":"ContainerStarted","Data":"e2bfe1093bf39b6079c816f77109a76c143e595328f461ea11efdb3ca78d71db"}
Oct 30 21:53:27.891081 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:27.891044    5061 generic.go:356] "Generic (PLEG): container finished" podID="147f4f98-ac6a-4b58-8750-a8410dd6324e" containerID="0691492afd2aebc1e5aa04014c2805dfd67ff4e3087e09d2d5a432f20c862d05" exitCode=0
Oct 30 21:53:27.891513 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:27.891122    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-chgcq" event={"ID":"147f4f98-ac6a-4b58-8750-a8410dd6324e","Type":"ContainerDied","Data":"0691492afd2aebc1e5aa04014c2805dfd67ff4e3087e09d2d5a432f20c862d05"}
Oct 30 21:53:27.891709 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:27.891690    5061 scope.go:122] "RemoveContainer" containerID="0691492afd2aebc1e5aa04014c2805dfd67ff4e3087e09d2d5a432f20c862d05"
Oct 30 21:53:27.893177 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:27.893151    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:53:27.893252 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:27.893226    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:53:27.893296 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:27.893257    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:53:28.978549 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:28.978390    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-chgcq" event={"ID":"147f4f98-ac6a-4b58-8750-a8410dd6324e","Type":"ContainerStarted","Data":"ec9430b1c8a6e0956573be159466e09daa706f3ceba7cafd80697c5747f4b8fc"}
Oct 30 21:53:29.143552 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:29.141992    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-chgcq"
Oct 30 21:53:29.143552 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:29.142315    5061 patch_prober.go:29] interesting pod/e2e-worker-w3-f776cb-openshift-default-74d577586f-chgcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Oct 30 21:53:29.143552 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:29.142373    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-chgcq" podUID="147f4f98-ac6a-4b58-8750-a8410dd6324e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Oct 30 21:53:29.432553 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:29.408016    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-17d809-686d4b9688-4zrv4"]
Oct 30 21:53:29.467552 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:29.453991    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-17d809-686d4b9688-4zrv4"
Oct 30 21:53:29.468026 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:29.455785    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-17d809-686d4b9688-4zrv4"]
Oct 30 21:53:29.562878 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:29.562833    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxxjq\" (UniqueName: \"kubernetes.io/projected/71fc13bb-d3e2-4a61-9942-4195b36d227c-kube-api-access-zxxjq\") pod \"payload-processing-e2e-worker-w4-17d809-686d4b9688-4zrv4\" (UID: \"71fc13bb-d3e2-4a61-9942-4195b36d227c\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-17d809-686d4b9688-4zrv4"
Oct 30 21:53:29.563090 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:29.562892    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71fc13bb-d3e2-4a61-9942-4195b36d227c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-17d809-686d4b9688-4zrv4\" (UID: \"71fc13bb-d3e2-4a61-9942-4195b36d227c\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-17d809-686d4b9688-4zrv4"
Oct 30 21:53:29.581600 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:29.580887    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-17d809\""
Oct 30 21:53:29.582939 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:29.582898    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-17d809-dockercfg-w5scv\""
Oct 30 21:53:29.664480 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:29.664123    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxxjq\" (UniqueName: \"kubernetes.io/projected/71fc13bb-d3e2-4a61-9942-4195b36d227c-kube-api-access-zxxjq\") pod \"payload-processing-e2e-worker-w4-17d809-686d4b9688-4zrv4\" (UID: \"71fc13bb-d3e2-4a61-9942-4195b36d227c\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-17d809-686d4b9688-4zrv4"
Oct 30 21:53:29.664799 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:29.664776    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71fc13bb-d3e2-4a61-9942-4195b36d227c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-17d809-686d4b9688-4zrv4\" (UID: \"71fc13bb-d3e2-4a61-9942-4195b36d227c\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-17d809-686d4b9688-4zrv4"
Oct 30 21:53:29.665749 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:29.665708    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71fc13bb-d3e2-4a61-9942-4195b36d227c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-17d809-686d4b9688-4zrv4\" (UID: \"71fc13bb-d3e2-4a61-9942-4195b36d227c\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-17d809-686d4b9688-4zrv4"
Oct 30 21:53:29.761695 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:29.761608    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxxjq\" (UniqueName: \"kubernetes.io/projected/71fc13bb-d3e2-4a61-9942-4195b36d227c-kube-api-access-zxxjq\") pod \"payload-processing-e2e-worker-w4-17d809-686d4b9688-4zrv4\" (UID: \"71fc13bb-d3e2-4a61-9942-4195b36d227c\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-17d809-686d4b9688-4zrv4"
Oct 30 21:53:29.797373 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:29.797345    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-17d809-686d4b9688-4zrv4"
Oct 30 21:53:29.860768 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:29.860734    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-5be0d1-6989644d59-xk2lk"
Oct 30 21:53:30.133552 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:30.129940    5061 patch_prober.go:29] interesting pod/e2e-worker-w3-f776cb-openshift-default-74d577586f-chgcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Oct 30 21:53:30.133552 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:30.129998    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-chgcq" podUID="147f4f98-ac6a-4b58-8750-a8410dd6324e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Oct 30 21:53:30.262553 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:30.262056    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-17d809-686d4b9688-4zrv4"]
Oct 30 21:53:30.266797 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:53:30.266763    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71fc13bb_d3e2_4a61_9942_4195b36d227c.slice/crio-653a8fa7ff5281ed23a02b154b13daafcde16c7fd9279de86a0ed18b45d4e1e5 WatchSource:0}: Error finding container 653a8fa7ff5281ed23a02b154b13daafcde16c7fd9279de86a0ed18b45d4e1e5: Status 404 returned error can't find the container with id 653a8fa7ff5281ed23a02b154b13daafcde16c7fd9279de86a0ed18b45d4e1e5
Oct 30 21:53:30.997856 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:30.997789    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-14af5c-bd596f55c-f4z8t" event={"ID":"4646779e-fc85-4860-adcf-9855d2d26fb1","Type":"ContainerStarted","Data":"f9d1f8f098f59de29fa76e16823e6a77e792a849e274d8366cb07eb7cb4e19be"}
Oct 30 21:53:30.998047 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:30.997886    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-14af5c-bd596f55c-f4z8t"
Oct 30 21:53:30.999408 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:30.999380    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-17d809-686d4b9688-4zrv4" event={"ID":"71fc13bb-d3e2-4a61-9942-4195b36d227c","Type":"ContainerStarted","Data":"80aa3195220ba7a419be23e77dac98eebdc4d67dde595d19e0af8354740743b9"}
Oct 30 21:53:30.999578 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:30.999419    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-17d809-686d4b9688-4zrv4" event={"ID":"71fc13bb-d3e2-4a61-9942-4195b36d227c","Type":"ContainerStarted","Data":"653a8fa7ff5281ed23a02b154b13daafcde16c7fd9279de86a0ed18b45d4e1e5"}
Oct 30 21:53:30.999578 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:30.999551    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-17d809-686d4b9688-4zrv4"
Oct 30 21:53:31.136549 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:31.131953    5061 patch_prober.go:29] interesting pod/e2e-worker-w3-f776cb-openshift-default-74d577586f-chgcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Oct 30 21:53:31.136549 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:31.132011    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-chgcq" podUID="147f4f98-ac6a-4b58-8750-a8410dd6324e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Oct 30 21:53:31.170629 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:31.163141    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-14af5c-bd596f55c-f4z8t" podStartSLOduration=2.121526783 podStartE2EDuration="7.163116575s" totalImagesPullingTime="5.041589792s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:53:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:53:31.131095558 +0000 UTC m=+664.743865317" watchObservedRunningTime="2026-10-30 21:53:31.163116575 +0000 UTC m=+664.775886334"
Oct 30 21:53:31.246595 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:31.246517    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-17d809-686d4b9688-4zrv4" podStartSLOduration=2.246495658 podStartE2EDuration="2.246495658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:53:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:53:31.244923392 +0000 UTC m=+664.857693150" watchObservedRunningTime="2026-10-30 21:53:31.246495658 +0000 UTC m=+664.859265417"
Oct 30 21:53:32.133552 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:32.130857    5061 patch_prober.go:29] interesting pod/e2e-worker-w3-f776cb-openshift-default-74d577586f-chgcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Oct 30 21:53:32.133552 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:32.130932    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-chgcq" podUID="147f4f98-ac6a-4b58-8750-a8410dd6324e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Oct 30 21:53:33.131551 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:33.130898    5061 patch_prober.go:29] interesting pod/e2e-worker-w3-f776cb-openshift-default-74d577586f-chgcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Oct 30 21:53:33.131551 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:33.130976    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-chgcq" podUID="147f4f98-ac6a-4b58-8750-a8410dd6324e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Oct 30 21:53:34.130001 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:34.129969    5061 patch_prober.go:29] interesting pod/e2e-worker-w3-f776cb-openshift-default-74d577586f-chgcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Oct 30 21:53:34.130189 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:34.130024    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-chgcq" podUID="147f4f98-ac6a-4b58-8750-a8410dd6324e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Oct 30 21:53:35.134627 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:35.134595    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-chgcq"
Oct 30 21:53:35.135143 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:35.134868    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-chgcq"
Oct 30 21:53:35.135721 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:35.135694    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-chgcq"
Oct 30 21:53:37.011067 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:37.011039    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-14af5c-bd596f55c-f4z8t"
Oct 30 21:53:38.076427 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:38.076394    5061 generic.go:356] "Generic (PLEG): container finished" podID="5eeacb87-9f34-4314-80f6-d538c5e843fa" containerID="e57f49eb55bc61870711f23aabe0f47acf4b5e58c65d60588e93417e4590abdf" exitCode=0
Oct 30 21:53:38.076893 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:38.076445    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-zskrq" event={"ID":"5eeacb87-9f34-4314-80f6-d538c5e843fa","Type":"ContainerDied","Data":"e57f49eb55bc61870711f23aabe0f47acf4b5e58c65d60588e93417e4590abdf"}
Oct 30 21:53:38.076893 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:38.076831    5061 scope.go:122] "RemoveContainer" containerID="e57f49eb55bc61870711f23aabe0f47acf4b5e58c65d60588e93417e4590abdf"
Oct 30 21:53:38.078313 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:38.078277    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:53:38.078423 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:38.078366    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:53:38.078423 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:38.078411    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:53:38.663119 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:38.663085    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-zskrq"
Oct 30 21:53:39.083655 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:39.083624    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-zskrq" event={"ID":"5eeacb87-9f34-4314-80f6-d538c5e843fa","Type":"ContainerStarted","Data":"14a3d242c861fb7cbe67ebdf86ad5e13a006edd2a1df573e31953065dc78cbbd"}
Oct 30 21:53:39.889993 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:39.889955    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-zskrq"
Oct 30 21:53:39.891194 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:39.891166    5061 patch_prober.go:29] interesting pod/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-zskrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Oct 30 21:53:39.891350 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:39.891218    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-zskrq" podUID="5eeacb87-9f34-4314-80f6-d538c5e843fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Oct 30 21:53:40.889950 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:40.889909    5061 patch_prober.go:29] interesting pod/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-zskrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Oct 30 21:53:40.890338 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:40.889980    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-zskrq" podUID="5eeacb87-9f34-4314-80f6-d538c5e843fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Oct 30 21:53:41.889937 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:41.889899    5061 patch_prober.go:29] interesting pod/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-zskrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Oct 30 21:53:41.890118 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:41.889959    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-zskrq" podUID="5eeacb87-9f34-4314-80f6-d538c5e843fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Oct 30 21:53:42.022839 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:42.022808    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-17d809-686d4b9688-4zrv4"
Oct 30 21:53:42.890558 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:42.890497    5061 patch_prober.go:29] interesting pod/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-zskrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Oct 30 21:53:42.891055 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:42.890566    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-zskrq" podUID="5eeacb87-9f34-4314-80f6-d538c5e843fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Oct 30 21:53:43.890167 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:43.890115    5061 patch_prober.go:29] interesting pod/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-zskrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Oct 30 21:53:43.890401 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:43.890190    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-zskrq" podUID="5eeacb87-9f34-4314-80f6-d538c5e843fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Oct 30 21:53:44.894347 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:44.894314    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-zskrq"
Oct 30 21:53:44.894755 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:44.894550    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-zskrq"
Oct 30 21:53:44.895161 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:53:44.895140    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-zskrq"
Oct 30 21:53:54.666271 ip-10-0-130-60 kubenswrapper[5061]: E1030 21:53:54.666233    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 21:54:08.126757 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:54:08.126721    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-2d1f91/simulator-w0-2d1f91-kserve-7699765c74-jwtkb"]
Oct 30 21:54:08.130984 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:54:08.130955    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2d1f91/simulator-w0-2d1f91-kserve-7699765c74-jwtkb"
Oct 30 21:54:08.228177 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:54:08.228139    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-2d1f91\"/\"simulator-w0-2d1f91-kserve-self-signed-certs\""
Oct 30 21:54:08.228329 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:54:08.228204    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-2d1f91\"/\"default-dockercfg-jv6g7\""
Oct 30 21:54:08.230748 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:54:08.230730    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-2d1f91/simulator-w0-2d1f91-kserve-7699765c74-jwtkb"]
Oct 30 21:54:08.234932 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:54:08.234910    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44b7d6f6-32d8-4ef4-95bc-eef604fa6b5b-tls-certs\") pod \"simulator-w0-2d1f91-kserve-7699765c74-jwtkb\" (UID: \"44b7d6f6-32d8-4ef4-95bc-eef604fa6b5b\") " pod="e2e-models-e2e-worker-w0-2d1f91/simulator-w0-2d1f91-kserve-7699765c74-jwtkb"
Oct 30 21:54:08.235037 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:54:08.234958    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44b7d6f6-32d8-4ef4-95bc-eef604fa6b5b-home\") pod \"simulator-w0-2d1f91-kserve-7699765c74-jwtkb\" (UID: \"44b7d6f6-32d8-4ef4-95bc-eef604fa6b5b\") " pod="e2e-models-e2e-worker-w0-2d1f91/simulator-w0-2d1f91-kserve-7699765c74-jwtkb"
Oct 30 21:54:08.235037 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:54:08.234991    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44b7d6f6-32d8-4ef4-95bc-eef604fa6b5b-tmp-dir\") pod \"simulator-w0-2d1f91-kserve-7699765c74-jwtkb\" (UID: \"44b7d6f6-32d8-4ef4-95bc-eef604fa6b5b\") " pod="e2e-models-e2e-worker-w0-2d1f91/simulator-w0-2d1f91-kserve-7699765c74-jwtkb"
Oct 30 21:54:08.235116 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:54:08.235049    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pg9wx\" (UniqueName: \"kubernetes.io/projected/44b7d6f6-32d8-4ef4-95bc-eef604fa6b5b-kube-api-access-pg9wx\") pod \"simulator-w0-2d1f91-kserve-7699765c74-jwtkb\" (UID: \"44b7d6f6-32d8-4ef4-95bc-eef604fa6b5b\") " pod="e2e-models-e2e-worker-w0-2d1f91/simulator-w0-2d1f91-kserve-7699765c74-jwtkb"
Oct 30 21:54:08.235196 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:54:08.235171    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44b7d6f6-32d8-4ef4-95bc-eef604fa6b5b-dshm\") pod \"simulator-w0-2d1f91-kserve-7699765c74-jwtkb\" (UID: \"44b7d6f6-32d8-4ef4-95bc-eef604fa6b5b\") " pod="e2e-models-e2e-worker-w0-2d1f91/simulator-w0-2d1f91-kserve-7699765c74-jwtkb"
Oct 30 21:54:08.235252 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:54:08.235240    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44b7d6f6-32d8-4ef4-95bc-eef604fa6b5b-model-cache\") pod \"simulator-w0-2d1f91-kserve-7699765c74-jwtkb\" (UID: \"44b7d6f6-32d8-4ef4-95bc-eef604fa6b5b\") " pod="e2e-models-e2e-worker-w0-2d1f91/simulator-w0-2d1f91-kserve-7699765c74-jwtkb"
Oct 30 21:54:08.251685 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:54:08.251657    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-2d1f91\"/\"openshift-service-ca.crt\""
Oct 30 21:54:08.251792 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:54:08.251769    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-2d1f91\"/\"kube-root-ca.crt\""
Oct 30 21:54:08.336021 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:54:08.335995    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44b7d6f6-32d8-4ef4-95bc-eef604fa6b5b-tls-certs\") pod \"simulator-w0-2d1f91-kserve-7699765c74-jwtkb\" (UID: \"44b7d6f6-32d8-4ef4-95bc-eef604fa6b5b\") " pod="e2e-models-e2e-worker-w0-2d1f91/simulator-w0-2d1f91-kserve-7699765c74-jwtkb"
Oct 30 21:54:08.336167 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:54:08.336037    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44b7d6f6-32d8-4ef4-95bc-eef604fa6b5b-home\") pod \"simulator-w0-2d1f91-kserve-7699765c74-jwtkb\" (UID: \"44b7d6f6-32d8-4ef4-95bc-eef604fa6b5b\") " pod="e2e-models-e2e-worker-w0-2d1f91/simulator-w0-2d1f91-kserve-7699765c74-jwtkb"
Oct 30 21:54:08.336167 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:54:08.336057    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44b7d6f6-32d8-4ef4-95bc-eef604fa6b5b-tmp-dir\") pod \"simulator-w0-2d1f91-kserve-7699765c74-jwtkb\" (UID: \"44b7d6f6-32d8-4ef4-95bc-eef604fa6b5b\") " pod="e2e-models-e2e-worker-w0-2d1f91/simulator-w0-2d1f91-kserve-7699765c74-jwtkb"
Oct 30 21:54:08.336167 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:54:08.336146    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pg9wx\" (UniqueName: \"kubernetes.io/projected/44b7d6f6-32d8-4ef4-95bc-eef604fa6b5b-kube-api-access-pg9wx\") pod \"simulator-w0-2d1f91-kserve-7699765c74-jwtkb\" (UID: \"44b7d6f6-32d8-4ef4-95bc-eef604fa6b5b\") " pod="e2e-models-e2e-worker-w0-2d1f91/simulator-w0-2d1f91-kserve-7699765c74-jwtkb"
Oct 30 21:54:08.336315 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:54:08.336175    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44b7d6f6-32d8-4ef4-95bc-eef604fa6b5b-dshm\") pod \"simulator-w0-2d1f91-kserve-7699765c74-jwtkb\" (UID: \"44b7d6f6-32d8-4ef4-95bc-eef604fa6b5b\") " pod="e2e-models-e2e-worker-w0-2d1f91/simulator-w0-2d1f91-kserve-7699765c74-jwtkb"
Oct 30 21:54:08.336315 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:54:08.336225    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44b7d6f6-32d8-4ef4-95bc-eef604fa6b5b-model-cache\") pod \"simulator-w0-2d1f91-kserve-7699765c74-jwtkb\" (UID: \"44b7d6f6-32d8-4ef4-95bc-eef604fa6b5b\") " pod="e2e-models-e2e-worker-w0-2d1f91/simulator-w0-2d1f91-kserve-7699765c74-jwtkb"
Oct 30 21:54:08.336496 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:54:08.336457    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44b7d6f6-32d8-4ef4-95bc-eef604fa6b5b-home\") pod \"simulator-w0-2d1f91-kserve-7699765c74-jwtkb\" (UID: \"44b7d6f6-32d8-4ef4-95bc-eef604fa6b5b\") " pod="e2e-models-e2e-worker-w0-2d1f91/simulator-w0-2d1f91-kserve-7699765c74-jwtkb"
Oct 30 21:54:08.336653 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:54:08.336634    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44b7d6f6-32d8-4ef4-95bc-eef604fa6b5b-model-cache\") pod \"simulator-w0-2d1f91-kserve-7699765c74-jwtkb\" (UID: \"44b7d6f6-32d8-4ef4-95bc-eef604fa6b5b\") " pod="e2e-models-e2e-worker-w0-2d1f91/simulator-w0-2d1f91-kserve-7699765c74-jwtkb"
Oct 30 21:54:08.336769 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:54:08.336743    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44b7d6f6-32d8-4ef4-95bc-eef604fa6b5b-tmp-dir\") pod \"simulator-w0-2d1f91-kserve-7699765c74-jwtkb\" (UID: \"44b7d6f6-32d8-4ef4-95bc-eef604fa6b5b\") " pod="e2e-models-e2e-worker-w0-2d1f91/simulator-w0-2d1f91-kserve-7699765c74-jwtkb"
Oct 30 21:54:08.338378 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:54:08.338356    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44b7d6f6-32d8-4ef4-95bc-eef604fa6b5b-dshm\") pod \"simulator-w0-2d1f91-kserve-7699765c74-jwtkb\" (UID: \"44b7d6f6-32d8-4ef4-95bc-eef604fa6b5b\") " pod="e2e-models-e2e-worker-w0-2d1f91/simulator-w0-2d1f91-kserve-7699765c74-jwtkb"
Oct 30 21:54:08.338378 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:54:08.338371    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44b7d6f6-32d8-4ef4-95bc-eef604fa6b5b-tls-certs\") pod \"simulator-w0-2d1f91-kserve-7699765c74-jwtkb\" (UID: \"44b7d6f6-32d8-4ef4-95bc-eef604fa6b5b\") " pod="e2e-models-e2e-worker-w0-2d1f91/simulator-w0-2d1f91-kserve-7699765c74-jwtkb"
Oct 30 21:54:08.394657 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:54:08.394596    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pg9wx\" (UniqueName: \"kubernetes.io/projected/44b7d6f6-32d8-4ef4-95bc-eef604fa6b5b-kube-api-access-pg9wx\") pod \"simulator-w0-2d1f91-kserve-7699765c74-jwtkb\" (UID: \"44b7d6f6-32d8-4ef4-95bc-eef604fa6b5b\") " pod="e2e-models-e2e-worker-w0-2d1f91/simulator-w0-2d1f91-kserve-7699765c74-jwtkb"
Oct 30 21:54:08.445011 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:54:08.444986    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2d1f91/simulator-w0-2d1f91-kserve-7699765c74-jwtkb"
Oct 30 21:54:08.701769 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:54:08.701679    5061 patch_prober.go:29] interesting pod/e2e-worker-w4-17d809-openshift-default-5b676976ff-ddh7p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:54:08.701769 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:54:08.701741    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-ddh7p" podUID="ba3aed70-33f3-4f0f-87cd-a8c19c347cd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:54:08.819464 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:54:08.819430    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-2d1f91/simulator-w0-2d1f91-kserve-7699765c74-jwtkb"]
Oct 30 21:54:08.825047 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:54:08.825023    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod44b7d6f6_32d8_4ef4_95bc_eef604fa6b5b.slice/crio-3617d361abd37f8a3cae63d267bedb2705258d6559fe26ef947385b3e38b58f2 WatchSource:0}: Error finding container 3617d361abd37f8a3cae63d267bedb2705258d6559fe26ef947385b3e38b58f2: Status 404 returned error can't find the container with id 3617d361abd37f8a3cae63d267bedb2705258d6559fe26ef947385b3e38b58f2
Oct 30 21:54:09.296487 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:54:09.296451    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2d1f91/simulator-w0-2d1f91-kserve-7699765c74-jwtkb" event={"ID":"44b7d6f6-32d8-4ef4-95bc-eef604fa6b5b","Type":"ContainerStarted","Data":"5d169ae8c76d09ccb1d3892d416f889929adf8f367b589272f0a20ec1d5e5c97"}
Oct 30 21:54:09.296876 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:54:09.296496    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2d1f91/simulator-w0-2d1f91-kserve-7699765c74-jwtkb" event={"ID":"44b7d6f6-32d8-4ef4-95bc-eef604fa6b5b","Type":"ContainerStarted","Data":"3617d361abd37f8a3cae63d267bedb2705258d6559fe26ef947385b3e38b58f2"}
Oct 30 21:54:09.408138 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:54:09.408087    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-2d1f91/simulator-w0-2d1f91-kserve-7699765c74-jwtkb" podStartSLOduration=1.408073825 podStartE2EDuration="1.408073825s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:09.406775744 +0000 UTC m=+703.019545504" watchObservedRunningTime="2026-10-30 21:54:09.408073825 +0000 UTC m=+703.020843582"
Oct 30 21:54:09.446089 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:54:09.446062    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2d1f91/simulator-w0-2d1f91-kserve-7699765c74-jwtkb"
Oct 30 21:54:10.303132 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:54:10.303097    5061 generic.go:356] "Generic (PLEG): container finished" podID="ba3aed70-33f3-4f0f-87cd-a8c19c347cd8" containerID="40776579af3656e34fb53d026a34a050339786ec5e6583ede8f3d72cf8844b91" exitCode=0
Oct 30 21:54:10.303676 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:54:10.303169    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-ddh7p" event={"ID":"ba3aed70-33f3-4f0f-87cd-a8c19c347cd8","Type":"ContainerDied","Data":"40776579af3656e34fb53d026a34a050339786ec5e6583ede8f3d72cf8844b91"}
Oct 30 21:54:10.303756 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:54:10.303678    5061 scope.go:122] "RemoveContainer" containerID="40776579af3656e34fb53d026a34a050339786ec5e6583ede8f3d72cf8844b91"
Oct 30 21:54:10.304920 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:54:10.304897    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:54:10.304982 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:54:10.304959    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:54:10.305017 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:54:10.304989    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:54:11.335582 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:54:11.335544    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-ddh7p" event={"ID":"ba3aed70-33f3-4f0f-87cd-a8c19c347cd8","Type":"ContainerStarted","Data":"680c40c3f9c87dea6ccec4aa32b8b20a014bb7104446bc4e25cf99ca1e61ea06"}
Oct 30 21:54:12.216767 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:54:12.216733    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-ddh7p"
Oct 30 21:54:12.217395 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:54:12.217368    5061 patch_prober.go:29] interesting pod/e2e-worker-w4-17d809-openshift-default-5b676976ff-ddh7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Oct 30 21:54:12.217604 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:54:12.217571    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-ddh7p" podUID="ba3aed70-33f3-4f0f-87cd-a8c19c347cd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Oct 30 21:54:13.229550 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:54:13.221637    5061 patch_prober.go:29] interesting pod/e2e-worker-w4-17d809-openshift-default-5b676976ff-ddh7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Oct 30 21:54:13.229550 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:54:13.221701    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-ddh7p" podUID="ba3aed70-33f3-4f0f-87cd-a8c19c347cd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Oct 30 21:54:14.220551 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:54:14.217832    5061 patch_prober.go:29] interesting pod/e2e-worker-w4-17d809-openshift-default-5b676976ff-ddh7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Oct 30 21:54:14.220551 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:54:14.217891    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-ddh7p" podUID="ba3aed70-33f3-4f0f-87cd-a8c19c347cd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Oct 30 21:54:15.222890 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:54:15.222849    5061 patch_prober.go:29] interesting pod/e2e-worker-w4-17d809-openshift-default-5b676976ff-ddh7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Oct 30 21:54:15.223481 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:54:15.223446    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-ddh7p" podUID="ba3aed70-33f3-4f0f-87cd-a8c19c347cd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Oct 30 21:54:16.228553 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:54:16.218821    5061 patch_prober.go:29] interesting pod/e2e-worker-w4-17d809-openshift-default-5b676976ff-ddh7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Oct 30 21:54:16.228553 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:54:16.218890    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-ddh7p" podUID="ba3aed70-33f3-4f0f-87cd-a8c19c347cd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Oct 30 21:54:17.219772 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:54:17.219743    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-ddh7p"
Oct 30 21:54:17.220048 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:54:17.220025    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-ddh7p"
Oct 30 21:54:17.220600 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:54:17.220585    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-ddh7p"
Oct 30 21:54:18.445152 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:54:18.445122    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-2d1f91/simulator-w0-2d1f91-kserve-7699765c74-jwtkb"
Oct 30 21:54:18.457922 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:54:18.457895    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-2d1f91/simulator-w0-2d1f91-kserve-7699765c74-jwtkb"
Oct 30 21:54:19.403940 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:54:19.403913    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2d1f91/simulator-w0-2d1f91-kserve-7699765c74-jwtkb"
Oct 30 21:54:24.666592 ip-10-0-130-60 kubenswrapper[5061]: E1030 21:54:24.666556    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 21:54:27.033570 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:54:27.033523    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-a07620/simulator-w1-a07620-kserve-55b44674f4-4z9lk"]
Oct 30 21:54:27.037473 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:54:27.037450    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a07620/simulator-w1-a07620-kserve-55b44674f4-4z9lk"
Oct 30 21:54:27.134893 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:54:27.134831    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-a07620\"/\"kube-root-ca.crt\""
Oct 30 21:54:27.135192 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:54:27.135163    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-a07620\"/\"simulator-w1-a07620-kserve-self-signed-certs\""
Oct 30 21:54:27.143562 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:54:27.135359    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-a07620\"/\"default-dockercfg-4gr9b\""
Oct 30 21:54:27.143562 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:54:27.135652    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-a07620\"/\"openshift-service-ca.crt\""
Oct 30 21:54:27.143562 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:54:27.142306    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-a07620/simulator-w1-a07620-kserve-55b44674f4-4z9lk"]
Oct 30 21:54:27.155163 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:54:27.155140    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4r2fq\" (UniqueName: \"kubernetes.io/projected/efe7c063-fb0e-4679-8a7c-13f2a0abaa74-kube-api-access-4r2fq\") pod \"simulator-w1-a07620-kserve-55b44674f4-4z9lk\" (UID: \"efe7c063-fb0e-4679-8a7c-13f2a0abaa74\") " pod="e2e-models-e2e-worker-w1-a07620/simulator-w1-a07620-kserve-55b44674f4-4z9lk"
Oct 30 21:54:27.155289 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:54:27.155191    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efe7c063-fb0e-4679-8a7c-13f2a0abaa74-tls-certs\") pod \"simulator-w1-a07620-kserve-55b44674f4-4z9lk\" (UID: \"efe7c063-fb0e-4679-8a7c-13f2a0abaa74\") " pod="e2e-models-e2e-worker-w1-a07620/simulator-w1-a07620-kserve-55b44674f4-4z9lk"
Oct 30 21:54:27.155332 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:54:27.155316    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/efe7c063-fb0e-4679-8a7c-13f2a0abaa74-dshm\") pod \"simulator-w1-a07620-kserve-55b44674f4-4z9lk\" (UID: \"efe7c063-fb0e-4679-8a7c-13f2a0abaa74\") " pod="e2e-models-e2e-worker-w1-a07620/simulator-w1-a07620-kserve-55b44674f4-4z9lk"
Oct 30 21:54:27.155367 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:54:27.155349    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/efe7c063-fb0e-4679-8a7c-13f2a0abaa74-tmp-dir\") pod \"simulator-w1-a07620-kserve-55b44674f4-4z9lk\" (UID: \"efe7c063-fb0e-4679-8a7c-13f2a0abaa74\") " pod="e2e-models-e2e-worker-w1-a07620/simulator-w1-a07620-kserve-55b44674f4-4z9lk"
Oct 30 21:54:27.155407 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:54:27.155366    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/efe7c063-fb0e-4679-8a7c-13f2a0abaa74-home\") pod \"simulator-w1-a07620-kserve-55b44674f4-4z9lk\" (UID: \"efe7c063-fb0e-4679-8a7c-13f2a0abaa74\") " pod="e2e-models-e2e-worker-w1-a07620/simulator-w1-a07620-kserve-55b44674f4-4z9lk"
Oct 30 21:54:27.155407 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:54:27.155382    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/efe7c063-fb0e-4679-8a7c-13f2a0abaa74-model-cache\") pod \"simulator-w1-a07620-kserve-55b44674f4-4z9lk\" (UID: \"efe7c063-fb0e-4679-8a7c-13f2a0abaa74\") " pod="e2e-models-e2e-worker-w1-a07620/simulator-w1-a07620-kserve-55b44674f4-4z9lk"
Oct 30 21:54:27.256959 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:54:27.256929    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/efe7c063-fb0e-4679-8a7c-13f2a0abaa74-dshm\") pod \"simulator-w1-a07620-kserve-55b44674f4-4z9lk\" (UID: \"efe7c063-fb0e-4679-8a7c-13f2a0abaa74\") " pod="e2e-models-e2e-worker-w1-a07620/simulator-w1-a07620-kserve-55b44674f4-4z9lk"
Oct 30 21:54:27.256959 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:54:27.256966    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/efe7c063-fb0e-4679-8a7c-13f2a0abaa74-tmp-dir\") pod \"simulator-w1-a07620-kserve-55b44674f4-4z9lk\" (UID: \"efe7c063-fb0e-4679-8a7c-13f2a0abaa74\") " pod="e2e-models-e2e-worker-w1-a07620/simulator-w1-a07620-kserve-55b44674f4-4z9lk"
Oct 30 21:54:27.257209 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:54:27.256982    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/efe7c063-fb0e-4679-8a7c-13f2a0abaa74-home\") pod \"simulator-w1-a07620-kserve-55b44674f4-4z9lk\" (UID: \"efe7c063-fb0e-4679-8a7c-13f2a0abaa74\") " pod="e2e-models-e2e-worker-w1-a07620/simulator-w1-a07620-kserve-55b44674f4-4z9lk"
Oct 30 21:54:27.257209 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:54:27.256998    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/efe7c063-fb0e-4679-8a7c-13f2a0abaa74-model-cache\") pod \"simulator-w1-a07620-kserve-55b44674f4-4z9lk\" (UID: \"efe7c063-fb0e-4679-8a7c-13f2a0abaa74\") " pod="e2e-models-e2e-worker-w1-a07620/simulator-w1-a07620-kserve-55b44674f4-4z9lk"
Oct 30 21:54:27.257209 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:54:27.257083    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4r2fq\" (UniqueName: \"kubernetes.io/projected/efe7c063-fb0e-4679-8a7c-13f2a0abaa74-kube-api-access-4r2fq\") pod \"simulator-w1-a07620-kserve-55b44674f4-4z9lk\" (UID: \"efe7c063-fb0e-4679-8a7c-13f2a0abaa74\") " pod="e2e-models-e2e-worker-w1-a07620/simulator-w1-a07620-kserve-55b44674f4-4z9lk"
Oct 30 21:54:27.257209 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:54:27.257147    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efe7c063-fb0e-4679-8a7c-13f2a0abaa74-tls-certs\") pod \"simulator-w1-a07620-kserve-55b44674f4-4z9lk\" (UID: \"efe7c063-fb0e-4679-8a7c-13f2a0abaa74\") " pod="e2e-models-e2e-worker-w1-a07620/simulator-w1-a07620-kserve-55b44674f4-4z9lk"
Oct 30 21:54:27.257411 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:54:27.257349    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/efe7c063-fb0e-4679-8a7c-13f2a0abaa74-tmp-dir\") pod \"simulator-w1-a07620-kserve-55b44674f4-4z9lk\" (UID: \"efe7c063-fb0e-4679-8a7c-13f2a0abaa74\") " pod="e2e-models-e2e-worker-w1-a07620/simulator-w1-a07620-kserve-55b44674f4-4z9lk"
Oct 30 21:54:27.257475 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:54:27.257410    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/efe7c063-fb0e-4679-8a7c-13f2a0abaa74-home\") pod \"simulator-w1-a07620-kserve-55b44674f4-4z9lk\" (UID: \"efe7c063-fb0e-4679-8a7c-13f2a0abaa74\") " pod="e2e-models-e2e-worker-w1-a07620/simulator-w1-a07620-kserve-55b44674f4-4z9lk"
Oct 30 21:54:27.257566 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:54:27.257541    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/efe7c063-fb0e-4679-8a7c-13f2a0abaa74-model-cache\") pod \"simulator-w1-a07620-kserve-55b44674f4-4z9lk\" (UID: \"efe7c063-fb0e-4679-8a7c-13f2a0abaa74\") " pod="e2e-models-e2e-worker-w1-a07620/simulator-w1-a07620-kserve-55b44674f4-4z9lk"
Oct 30 21:54:27.259363 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:54:27.259344    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/efe7c063-fb0e-4679-8a7c-13f2a0abaa74-dshm\") pod \"simulator-w1-a07620-kserve-55b44674f4-4z9lk\" (UID: \"efe7c063-fb0e-4679-8a7c-13f2a0abaa74\") " pod="e2e-models-e2e-worker-w1-a07620/simulator-w1-a07620-kserve-55b44674f4-4z9lk"
Oct 30 21:54:27.259556 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:54:27.259522    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efe7c063-fb0e-4679-8a7c-13f2a0abaa74-tls-certs\") pod \"simulator-w1-a07620-kserve-55b44674f4-4z9lk\" (UID: \"efe7c063-fb0e-4679-8a7c-13f2a0abaa74\") " pod="e2e-models-e2e-worker-w1-a07620/simulator-w1-a07620-kserve-55b44674f4-4z9lk"
Oct 30 21:54:27.311294 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:54:27.311261    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4r2fq\" (UniqueName: \"kubernetes.io/projected/efe7c063-fb0e-4679-8a7c-13f2a0abaa74-kube-api-access-4r2fq\") pod \"simulator-w1-a07620-kserve-55b44674f4-4z9lk\" (UID: \"efe7c063-fb0e-4679-8a7c-13f2a0abaa74\") " pod="e2e-models-e2e-worker-w1-a07620/simulator-w1-a07620-kserve-55b44674f4-4z9lk"
Oct 30 21:54:27.349992 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:54:27.349966    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a07620/simulator-w1-a07620-kserve-55b44674f4-4z9lk"
Oct 30 21:54:27.720063 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:54:27.719945    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-a07620/simulator-w1-a07620-kserve-55b44674f4-4z9lk"]
Oct 30 21:54:27.724998 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:54:27.724965    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podefe7c063_fb0e_4679_8a7c_13f2a0abaa74.slice/crio-a0e4fbd8ee4d11a80962c68d4c12d5f3fa1d364f6aba042490ad79e80412fbd2 WatchSource:0}: Error finding container a0e4fbd8ee4d11a80962c68d4c12d5f3fa1d364f6aba042490ad79e80412fbd2: Status 404 returned error can't find the container with id a0e4fbd8ee4d11a80962c68d4c12d5f3fa1d364f6aba042490ad79e80412fbd2
Oct 30 21:54:28.447597 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:54:28.447559    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a07620/simulator-w1-a07620-kserve-55b44674f4-4z9lk" event={"ID":"efe7c063-fb0e-4679-8a7c-13f2a0abaa74","Type":"ContainerStarted","Data":"0feba69f7cea4a1d44ba150288760dd74da547313ff80433eec1bf73c2b9c466"}
Oct 30 21:54:28.447597 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:54:28.447601    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a07620/simulator-w1-a07620-kserve-55b44674f4-4z9lk" event={"ID":"efe7c063-fb0e-4679-8a7c-13f2a0abaa74","Type":"ContainerStarted","Data":"a0e4fbd8ee4d11a80962c68d4c12d5f3fa1d364f6aba042490ad79e80412fbd2"}
Oct 30 21:54:28.560468 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:54:28.560418    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-a07620/simulator-w1-a07620-kserve-55b44674f4-4z9lk" podStartSLOduration=2.560405479 podStartE2EDuration="2.560405479s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:28.558891596 +0000 UTC m=+722.171661353" watchObservedRunningTime="2026-10-30 21:54:28.560405479 +0000 UTC m=+722.173175237"
Oct 30 21:54:29.350068 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:54:29.350040    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a07620/simulator-w1-a07620-kserve-55b44674f4-4z9lk"
Oct 30 21:54:37.350735 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:54:37.350698    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-a07620/simulator-w1-a07620-kserve-55b44674f4-4z9lk"
Oct 30 21:54:37.363338 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:54:37.363306    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-a07620/simulator-w1-a07620-kserve-55b44674f4-4z9lk"
Oct 30 21:54:37.514567 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:54:37.514514    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a07620/simulator-w1-a07620-kserve-55b44674f4-4z9lk"
Oct 30 21:54:40.895875 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:54:40.895834    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-14af5c/simulator-w6-14af5c-kserve-85b8bd96d7-wlcb9"]
Oct 30 21:54:40.901308 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:54:40.901283    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-14af5c/simulator-w6-14af5c-kserve-85b8bd96d7-wlcb9"
Oct 30 21:54:40.998823 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:54:40.998790    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-14af5c\"/\"default-dockercfg-dzvk8\""
Oct 30 21:54:40.999015 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:54:40.998850    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-14af5c\"/\"kube-root-ca.crt\""
Oct 30 21:54:40.999015 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:54:40.998860    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-14af5c\"/\"openshift-service-ca.crt\""
Oct 30 21:54:40.999015 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:54:40.998902    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-14af5c\"/\"simulator-w6-14af5c-kserve-self-signed-certs\""
Oct 30 21:54:41.004308 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:54:41.004272    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-14af5c/simulator-w6-14af5c-kserve-85b8bd96d7-wlcb9"]
Oct 30 21:54:41.010707 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:54:41.010677    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39766c7d-5f1d-42db-8920-c0321cd12c99-dshm\") pod \"simulator-w6-14af5c-kserve-85b8bd96d7-wlcb9\" (UID: \"39766c7d-5f1d-42db-8920-c0321cd12c99\") " pod="e2e-models-e2e-worker-w6-14af5c/simulator-w6-14af5c-kserve-85b8bd96d7-wlcb9"
Oct 30 21:54:41.010858 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:54:41.010717    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39766c7d-5f1d-42db-8920-c0321cd12c99-model-cache\") pod \"simulator-w6-14af5c-kserve-85b8bd96d7-wlcb9\" (UID: \"39766c7d-5f1d-42db-8920-c0321cd12c99\") " pod="e2e-models-e2e-worker-w6-14af5c/simulator-w6-14af5c-kserve-85b8bd96d7-wlcb9"
Oct 30 21:54:41.010858 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:54:41.010741    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rcgj\" (UniqueName: \"kubernetes.io/projected/39766c7d-5f1d-42db-8920-c0321cd12c99-kube-api-access-5rcgj\") pod \"simulator-w6-14af5c-kserve-85b8bd96d7-wlcb9\" (UID: \"39766c7d-5f1d-42db-8920-c0321cd12c99\") " pod="e2e-models-e2e-worker-w6-14af5c/simulator-w6-14af5c-kserve-85b8bd96d7-wlcb9"
Oct 30 21:54:41.010974 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:54:41.010881    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39766c7d-5f1d-42db-8920-c0321cd12c99-home\") pod \"simulator-w6-14af5c-kserve-85b8bd96d7-wlcb9\" (UID: \"39766c7d-5f1d-42db-8920-c0321cd12c99\") " pod="e2e-models-e2e-worker-w6-14af5c/simulator-w6-14af5c-kserve-85b8bd96d7-wlcb9"
Oct 30 21:54:41.011040 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:54:41.011021    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39766c7d-5f1d-42db-8920-c0321cd12c99-tls-certs\") pod \"simulator-w6-14af5c-kserve-85b8bd96d7-wlcb9\" (UID: \"39766c7d-5f1d-42db-8920-c0321cd12c99\") " pod="e2e-models-e2e-worker-w6-14af5c/simulator-w6-14af5c-kserve-85b8bd96d7-wlcb9"
Oct 30 21:54:41.011100 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:54:41.011049    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39766c7d-5f1d-42db-8920-c0321cd12c99-tmp-dir\") pod \"simulator-w6-14af5c-kserve-85b8bd96d7-wlcb9\" (UID: \"39766c7d-5f1d-42db-8920-c0321cd12c99\") " pod="e2e-models-e2e-worker-w6-14af5c/simulator-w6-14af5c-kserve-85b8bd96d7-wlcb9"
Oct 30 21:54:41.111902 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:54:41.111866    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39766c7d-5f1d-42db-8920-c0321cd12c99-dshm\") pod \"simulator-w6-14af5c-kserve-85b8bd96d7-wlcb9\" (UID: \"39766c7d-5f1d-42db-8920-c0321cd12c99\") " pod="e2e-models-e2e-worker-w6-14af5c/simulator-w6-14af5c-kserve-85b8bd96d7-wlcb9"
Oct 30 21:54:41.112132 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:54:41.111916    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39766c7d-5f1d-42db-8920-c0321cd12c99-model-cache\") pod \"simulator-w6-14af5c-kserve-85b8bd96d7-wlcb9\" (UID: \"39766c7d-5f1d-42db-8920-c0321cd12c99\") " pod="e2e-models-e2e-worker-w6-14af5c/simulator-w6-14af5c-kserve-85b8bd96d7-wlcb9"
Oct 30 21:54:41.112132 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:54:41.112041    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rcgj\" (UniqueName: \"kubernetes.io/projected/39766c7d-5f1d-42db-8920-c0321cd12c99-kube-api-access-5rcgj\") pod \"simulator-w6-14af5c-kserve-85b8bd96d7-wlcb9\" (UID: \"39766c7d-5f1d-42db-8920-c0321cd12c99\") " pod="e2e-models-e2e-worker-w6-14af5c/simulator-w6-14af5c-kserve-85b8bd96d7-wlcb9"
Oct 30 21:54:41.112132 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:54:41.112114    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39766c7d-5f1d-42db-8920-c0321cd12c99-home\") pod \"simulator-w6-14af5c-kserve-85b8bd96d7-wlcb9\" (UID: \"39766c7d-5f1d-42db-8920-c0321cd12c99\") " pod="e2e-models-e2e-worker-w6-14af5c/simulator-w6-14af5c-kserve-85b8bd96d7-wlcb9"
Oct 30 21:54:41.112306 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:54:41.112288    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39766c7d-5f1d-42db-8920-c0321cd12c99-tls-certs\") pod \"simulator-w6-14af5c-kserve-85b8bd96d7-wlcb9\" (UID: \"39766c7d-5f1d-42db-8920-c0321cd12c99\") " pod="e2e-models-e2e-worker-w6-14af5c/simulator-w6-14af5c-kserve-85b8bd96d7-wlcb9"
Oct 30 21:54:41.112364 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:54:41.112317    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39766c7d-5f1d-42db-8920-c0321cd12c99-tmp-dir\") pod \"simulator-w6-14af5c-kserve-85b8bd96d7-wlcb9\" (UID: \"39766c7d-5f1d-42db-8920-c0321cd12c99\") " pod="e2e-models-e2e-worker-w6-14af5c/simulator-w6-14af5c-kserve-85b8bd96d7-wlcb9"
Oct 30 21:54:41.112580 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:54:41.112560    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39766c7d-5f1d-42db-8920-c0321cd12c99-model-cache\") pod \"simulator-w6-14af5c-kserve-85b8bd96d7-wlcb9\" (UID: \"39766c7d-5f1d-42db-8920-c0321cd12c99\") " pod="e2e-models-e2e-worker-w6-14af5c/simulator-w6-14af5c-kserve-85b8bd96d7-wlcb9"
Oct 30 21:54:41.112692 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:54:41.112626    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39766c7d-5f1d-42db-8920-c0321cd12c99-home\") pod \"simulator-w6-14af5c-kserve-85b8bd96d7-wlcb9\" (UID: \"39766c7d-5f1d-42db-8920-c0321cd12c99\") " pod="e2e-models-e2e-worker-w6-14af5c/simulator-w6-14af5c-kserve-85b8bd96d7-wlcb9"
Oct 30 21:54:41.112763 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:54:41.112731    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39766c7d-5f1d-42db-8920-c0321cd12c99-tmp-dir\") pod \"simulator-w6-14af5c-kserve-85b8bd96d7-wlcb9\" (UID: \"39766c7d-5f1d-42db-8920-c0321cd12c99\") " pod="e2e-models-e2e-worker-w6-14af5c/simulator-w6-14af5c-kserve-85b8bd96d7-wlcb9"
Oct 30 21:54:41.114411 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:54:41.114391    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39766c7d-5f1d-42db-8920-c0321cd12c99-dshm\") pod \"simulator-w6-14af5c-kserve-85b8bd96d7-wlcb9\" (UID: \"39766c7d-5f1d-42db-8920-c0321cd12c99\") " pod="e2e-models-e2e-worker-w6-14af5c/simulator-w6-14af5c-kserve-85b8bd96d7-wlcb9"
Oct 30 21:54:41.115732 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:54:41.115710    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39766c7d-5f1d-42db-8920-c0321cd12c99-tls-certs\") pod \"simulator-w6-14af5c-kserve-85b8bd96d7-wlcb9\" (UID: \"39766c7d-5f1d-42db-8920-c0321cd12c99\") " pod="e2e-models-e2e-worker-w6-14af5c/simulator-w6-14af5c-kserve-85b8bd96d7-wlcb9"
Oct 30 21:54:41.170398 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:54:41.170329    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rcgj\" (UniqueName: \"kubernetes.io/projected/39766c7d-5f1d-42db-8920-c0321cd12c99-kube-api-access-5rcgj\") pod \"simulator-w6-14af5c-kserve-85b8bd96d7-wlcb9\" (UID: \"39766c7d-5f1d-42db-8920-c0321cd12c99\") " pod="e2e-models-e2e-worker-w6-14af5c/simulator-w6-14af5c-kserve-85b8bd96d7-wlcb9"
Oct 30 21:54:41.213856 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:54:41.213818    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-14af5c/simulator-w6-14af5c-kserve-85b8bd96d7-wlcb9"
Oct 30 21:54:41.587628 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:54:41.587589    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-14af5c/simulator-w6-14af5c-kserve-85b8bd96d7-wlcb9"]
Oct 30 21:54:41.592440 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:54:41.592412    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod39766c7d_5f1d_42db_8920_c0321cd12c99.slice/crio-5ff6dfd897ec36965388217906652f2a8c6347d1b0aa2a64f19b0a947cf04a72 WatchSource:0}: Error finding container 5ff6dfd897ec36965388217906652f2a8c6347d1b0aa2a64f19b0a947cf04a72: Status 404 returned error can't find the container with id 5ff6dfd897ec36965388217906652f2a8c6347d1b0aa2a64f19b0a947cf04a72
Oct 30 21:54:42.532266 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:54:42.532228    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-14af5c/simulator-w6-14af5c-kserve-85b8bd96d7-wlcb9" event={"ID":"39766c7d-5f1d-42db-8920-c0321cd12c99","Type":"ContainerStarted","Data":"4742be0ecaf01e860e10e4751e0a2620e7d66623622e6cc0756908c05a65b260"}
Oct 30 21:54:42.532266 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:54:42.532268    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-14af5c/simulator-w6-14af5c-kserve-85b8bd96d7-wlcb9" event={"ID":"39766c7d-5f1d-42db-8920-c0321cd12c99","Type":"ContainerStarted","Data":"5ff6dfd897ec36965388217906652f2a8c6347d1b0aa2a64f19b0a947cf04a72"}
Oct 30 21:54:42.643931 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:54:42.643882    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-14af5c/simulator-w6-14af5c-kserve-85b8bd96d7-wlcb9" podStartSLOduration=2.643868437 podStartE2EDuration="2.643868437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:42.642572693 +0000 UTC m=+736.255342451" watchObservedRunningTime="2026-10-30 21:54:42.643868437 +0000 UTC m=+736.256638197"
Oct 30 21:54:43.214644 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:54:43.214613    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-14af5c/simulator-w6-14af5c-kserve-85b8bd96d7-wlcb9"
Oct 30 21:54:51.214084 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:54:51.214049    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-14af5c/simulator-w6-14af5c-kserve-85b8bd96d7-wlcb9"
Oct 30 21:54:51.227670 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:54:51.227643    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-14af5c/simulator-w6-14af5c-kserve-85b8bd96d7-wlcb9"
Oct 30 21:54:51.602065 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:54:51.602030    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-14af5c/simulator-w6-14af5c-kserve-85b8bd96d7-wlcb9"
Oct 30 21:54:54.667019 ip-10-0-130-60 kubenswrapper[5061]: E1030 21:54:54.666980    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 21:55:19.662347 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:55:19.662265    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-17d809/simulator-w4-17d809-kserve-78d67b6f4-7hsvb"]
Oct 30 21:55:19.666795 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:55:19.666771    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-17d809/simulator-w4-17d809-kserve-78d67b6f4-7hsvb"
Oct 30 21:55:19.735878 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:55:19.735847    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7337cf8-4f59-4afb-9b50-6b6c9fa4a773-dshm\") pod \"simulator-w4-17d809-kserve-78d67b6f4-7hsvb\" (UID: \"e7337cf8-4f59-4afb-9b50-6b6c9fa4a773\") " pod="e2e-models-e2e-worker-w4-17d809/simulator-w4-17d809-kserve-78d67b6f4-7hsvb"
Oct 30 21:55:19.735878 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:55:19.735880    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7337cf8-4f59-4afb-9b50-6b6c9fa4a773-tls-certs\") pod \"simulator-w4-17d809-kserve-78d67b6f4-7hsvb\" (UID: \"e7337cf8-4f59-4afb-9b50-6b6c9fa4a773\") " pod="e2e-models-e2e-worker-w4-17d809/simulator-w4-17d809-kserve-78d67b6f4-7hsvb"
Oct 30 21:55:19.736113 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:55:19.735911    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7337cf8-4f59-4afb-9b50-6b6c9fa4a773-tmp-dir\") pod \"simulator-w4-17d809-kserve-78d67b6f4-7hsvb\" (UID: \"e7337cf8-4f59-4afb-9b50-6b6c9fa4a773\") " pod="e2e-models-e2e-worker-w4-17d809/simulator-w4-17d809-kserve-78d67b6f4-7hsvb"
Oct 30 21:55:19.736113 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:55:19.736086    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7337cf8-4f59-4afb-9b50-6b6c9fa4a773-model-cache\") pod \"simulator-w4-17d809-kserve-78d67b6f4-7hsvb\" (UID: \"e7337cf8-4f59-4afb-9b50-6b6c9fa4a773\") " pod="e2e-models-e2e-worker-w4-17d809/simulator-w4-17d809-kserve-78d67b6f4-7hsvb"
Oct 30 21:55:19.736227 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:55:19.736132    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ngwt\" (UniqueName: \"kubernetes.io/projected/e7337cf8-4f59-4afb-9b50-6b6c9fa4a773-kube-api-access-9ngwt\") pod \"simulator-w4-17d809-kserve-78d67b6f4-7hsvb\" (UID: \"e7337cf8-4f59-4afb-9b50-6b6c9fa4a773\") " pod="e2e-models-e2e-worker-w4-17d809/simulator-w4-17d809-kserve-78d67b6f4-7hsvb"
Oct 30 21:55:19.736288 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:55:19.736224    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7337cf8-4f59-4afb-9b50-6b6c9fa4a773-home\") pod \"simulator-w4-17d809-kserve-78d67b6f4-7hsvb\" (UID: \"e7337cf8-4f59-4afb-9b50-6b6c9fa4a773\") " pod="e2e-models-e2e-worker-w4-17d809/simulator-w4-17d809-kserve-78d67b6f4-7hsvb"
Oct 30 21:55:19.764326 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:55:19.764305    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-17d809\"/\"openshift-service-ca.crt\""
Oct 30 21:55:19.764326 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:55:19.764305    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-17d809\"/\"simulator-w4-17d809-kserve-self-signed-certs\""
Oct 30 21:55:19.764490 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:55:19.764329    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-17d809\"/\"default-dockercfg-v7d58\""
Oct 30 21:55:19.764490 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:55:19.764369    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-17d809\"/\"kube-root-ca.crt\""
Oct 30 21:55:19.767006 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:55:19.766988    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-17d809/simulator-w4-17d809-kserve-78d67b6f4-7hsvb"]
Oct 30 21:55:19.837117 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:55:19.837085    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7337cf8-4f59-4afb-9b50-6b6c9fa4a773-tmp-dir\") pod \"simulator-w4-17d809-kserve-78d67b6f4-7hsvb\" (UID: \"e7337cf8-4f59-4afb-9b50-6b6c9fa4a773\") " pod="e2e-models-e2e-worker-w4-17d809/simulator-w4-17d809-kserve-78d67b6f4-7hsvb"
Oct 30 21:55:19.837316 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:55:19.837166    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7337cf8-4f59-4afb-9b50-6b6c9fa4a773-model-cache\") pod \"simulator-w4-17d809-kserve-78d67b6f4-7hsvb\" (UID: \"e7337cf8-4f59-4afb-9b50-6b6c9fa4a773\") " pod="e2e-models-e2e-worker-w4-17d809/simulator-w4-17d809-kserve-78d67b6f4-7hsvb"
Oct 30 21:55:19.837316 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:55:19.837204    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ngwt\" (UniqueName: \"kubernetes.io/projected/e7337cf8-4f59-4afb-9b50-6b6c9fa4a773-kube-api-access-9ngwt\") pod \"simulator-w4-17d809-kserve-78d67b6f4-7hsvb\" (UID: \"e7337cf8-4f59-4afb-9b50-6b6c9fa4a773\") " pod="e2e-models-e2e-worker-w4-17d809/simulator-w4-17d809-kserve-78d67b6f4-7hsvb"
Oct 30 21:55:19.837316 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:55:19.837235    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7337cf8-4f59-4afb-9b50-6b6c9fa4a773-home\") pod \"simulator-w4-17d809-kserve-78d67b6f4-7hsvb\" (UID: \"e7337cf8-4f59-4afb-9b50-6b6c9fa4a773\") " pod="e2e-models-e2e-worker-w4-17d809/simulator-w4-17d809-kserve-78d67b6f4-7hsvb"
Oct 30 21:55:19.837316 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:55:19.837291    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7337cf8-4f59-4afb-9b50-6b6c9fa4a773-dshm\") pod \"simulator-w4-17d809-kserve-78d67b6f4-7hsvb\" (UID: \"e7337cf8-4f59-4afb-9b50-6b6c9fa4a773\") " pod="e2e-models-e2e-worker-w4-17d809/simulator-w4-17d809-kserve-78d67b6f4-7hsvb"
Oct 30 21:55:19.837316 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:55:19.837308    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7337cf8-4f59-4afb-9b50-6b6c9fa4a773-tls-certs\") pod \"simulator-w4-17d809-kserve-78d67b6f4-7hsvb\" (UID: \"e7337cf8-4f59-4afb-9b50-6b6c9fa4a773\") " pod="e2e-models-e2e-worker-w4-17d809/simulator-w4-17d809-kserve-78d67b6f4-7hsvb"
Oct 30 21:55:19.837624 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:55:19.837601    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7337cf8-4f59-4afb-9b50-6b6c9fa4a773-tmp-dir\") pod \"simulator-w4-17d809-kserve-78d67b6f4-7hsvb\" (UID: \"e7337cf8-4f59-4afb-9b50-6b6c9fa4a773\") " pod="e2e-models-e2e-worker-w4-17d809/simulator-w4-17d809-kserve-78d67b6f4-7hsvb"
Oct 30 21:55:19.837678 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:55:19.837633    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7337cf8-4f59-4afb-9b50-6b6c9fa4a773-model-cache\") pod \"simulator-w4-17d809-kserve-78d67b6f4-7hsvb\" (UID: \"e7337cf8-4f59-4afb-9b50-6b6c9fa4a773\") " pod="e2e-models-e2e-worker-w4-17d809/simulator-w4-17d809-kserve-78d67b6f4-7hsvb"
Oct 30 21:55:19.837718 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:55:19.837692    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7337cf8-4f59-4afb-9b50-6b6c9fa4a773-home\") pod \"simulator-w4-17d809-kserve-78d67b6f4-7hsvb\" (UID: \"e7337cf8-4f59-4afb-9b50-6b6c9fa4a773\") " pod="e2e-models-e2e-worker-w4-17d809/simulator-w4-17d809-kserve-78d67b6f4-7hsvb"
Oct 30 21:55:19.839610 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:55:19.839594    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7337cf8-4f59-4afb-9b50-6b6c9fa4a773-dshm\") pod \"simulator-w4-17d809-kserve-78d67b6f4-7hsvb\" (UID: \"e7337cf8-4f59-4afb-9b50-6b6c9fa4a773\") " pod="e2e-models-e2e-worker-w4-17d809/simulator-w4-17d809-kserve-78d67b6f4-7hsvb"
Oct 30 21:55:19.840013 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:55:19.839992    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7337cf8-4f59-4afb-9b50-6b6c9fa4a773-tls-certs\") pod \"simulator-w4-17d809-kserve-78d67b6f4-7hsvb\" (UID: \"e7337cf8-4f59-4afb-9b50-6b6c9fa4a773\") " pod="e2e-models-e2e-worker-w4-17d809/simulator-w4-17d809-kserve-78d67b6f4-7hsvb"
Oct 30 21:55:19.893325 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:55:19.893295    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ngwt\" (UniqueName: \"kubernetes.io/projected/e7337cf8-4f59-4afb-9b50-6b6c9fa4a773-kube-api-access-9ngwt\") pod \"simulator-w4-17d809-kserve-78d67b6f4-7hsvb\" (UID: \"e7337cf8-4f59-4afb-9b50-6b6c9fa4a773\") " pod="e2e-models-e2e-worker-w4-17d809/simulator-w4-17d809-kserve-78d67b6f4-7hsvb"
Oct 30 21:55:19.980098 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:55:19.980023    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-17d809/simulator-w4-17d809-kserve-78d67b6f4-7hsvb"
Oct 30 21:55:20.358103 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:55:20.358066    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-17d809/simulator-w4-17d809-kserve-78d67b6f4-7hsvb"]
Oct 30 21:55:20.362849 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:55:20.362824    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode7337cf8_4f59_4afb_9b50_6b6c9fa4a773.slice/crio-3537553b37ce1a1823f026ffa7bc4e058173b3cb104f4591e5f77130757be781 WatchSource:0}: Error finding container 3537553b37ce1a1823f026ffa7bc4e058173b3cb104f4591e5f77130757be781: Status 404 returned error can't find the container with id 3537553b37ce1a1823f026ffa7bc4e058173b3cb104f4591e5f77130757be781
Oct 30 21:55:20.764569 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:55:20.764474    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-17d809/simulator-w4-17d809-kserve-78d67b6f4-7hsvb" event={"ID":"e7337cf8-4f59-4afb-9b50-6b6c9fa4a773","Type":"ContainerStarted","Data":"92fd7af56c5fb28daced7d6a9e574f44e4eed09cf90c8e4d8851bc2a78c11333"}
Oct 30 21:55:20.764569 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:55:20.764516    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-17d809/simulator-w4-17d809-kserve-78d67b6f4-7hsvb" event={"ID":"e7337cf8-4f59-4afb-9b50-6b6c9fa4a773","Type":"ContainerStarted","Data":"3537553b37ce1a1823f026ffa7bc4e058173b3cb104f4591e5f77130757be781"}
Oct 30 21:55:20.878166 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:55:20.878118    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-17d809/simulator-w4-17d809-kserve-78d67b6f4-7hsvb" podStartSLOduration=1.87810598 podStartE2EDuration="1.87810598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:55:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:55:20.875674816 +0000 UTC m=+774.488444574" watchObservedRunningTime="2026-10-30 21:55:20.87810598 +0000 UTC m=+774.490875738"
Oct 30 21:55:20.980483 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:55:20.980456    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-17d809/simulator-w4-17d809-kserve-78d67b6f4-7hsvb"
Oct 30 21:55:23.783296 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:55:23.783262    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-2d1f91/premium-w0-2d1f91-kserve-6965b558cf-59lgw"]
Oct 30 21:55:23.787196 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:55:23.787177    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2d1f91/premium-w0-2d1f91-kserve-6965b558cf-59lgw"
Oct 30 21:55:23.877663 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:55:23.877630    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb84233d-25e0-48bc-8899-c3a3cb3cfeb9-tls-certs\") pod \"premium-w0-2d1f91-kserve-6965b558cf-59lgw\" (UID: \"fb84233d-25e0-48bc-8899-c3a3cb3cfeb9\") " pod="e2e-models-e2e-worker-w0-2d1f91/premium-w0-2d1f91-kserve-6965b558cf-59lgw"
Oct 30 21:55:23.877851 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:55:23.877702    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb84233d-25e0-48bc-8899-c3a3cb3cfeb9-model-cache\") pod \"premium-w0-2d1f91-kserve-6965b558cf-59lgw\" (UID: \"fb84233d-25e0-48bc-8899-c3a3cb3cfeb9\") " pod="e2e-models-e2e-worker-w0-2d1f91/premium-w0-2d1f91-kserve-6965b558cf-59lgw"
Oct 30 21:55:23.877851 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:55:23.877781    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8z9h2\" (UniqueName: \"kubernetes.io/projected/fb84233d-25e0-48bc-8899-c3a3cb3cfeb9-kube-api-access-8z9h2\") pod \"premium-w0-2d1f91-kserve-6965b558cf-59lgw\" (UID: \"fb84233d-25e0-48bc-8899-c3a3cb3cfeb9\") " pod="e2e-models-e2e-worker-w0-2d1f91/premium-w0-2d1f91-kserve-6965b558cf-59lgw"
Oct 30 21:55:23.877851 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:55:23.877812    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb84233d-25e0-48bc-8899-c3a3cb3cfeb9-tmp-dir\") pod \"premium-w0-2d1f91-kserve-6965b558cf-59lgw\" (UID: \"fb84233d-25e0-48bc-8899-c3a3cb3cfeb9\") " pod="e2e-models-e2e-worker-w0-2d1f91/premium-w0-2d1f91-kserve-6965b558cf-59lgw"
Oct 30 21:55:23.878035 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:55:23.877917    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb84233d-25e0-48bc-8899-c3a3cb3cfeb9-dshm\") pod \"premium-w0-2d1f91-kserve-6965b558cf-59lgw\" (UID: \"fb84233d-25e0-48bc-8899-c3a3cb3cfeb9\") " pod="e2e-models-e2e-worker-w0-2d1f91/premium-w0-2d1f91-kserve-6965b558cf-59lgw"
Oct 30 21:55:23.878035 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:55:23.877952    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb84233d-25e0-48bc-8899-c3a3cb3cfeb9-home\") pod \"premium-w0-2d1f91-kserve-6965b558cf-59lgw\" (UID: \"fb84233d-25e0-48bc-8899-c3a3cb3cfeb9\") " pod="e2e-models-e2e-worker-w0-2d1f91/premium-w0-2d1f91-kserve-6965b558cf-59lgw"
Oct 30 21:55:23.884637 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:55:23.884614    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-2d1f91\"/\"premium-w0-2d1f91-kserve-self-signed-certs\""
Oct 30 21:55:23.887197 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:55:23.887178    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-2d1f91/premium-w0-2d1f91-kserve-6965b558cf-59lgw"]
Oct 30 21:55:23.979606 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:55:23.979573    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb84233d-25e0-48bc-8899-c3a3cb3cfeb9-home\") pod \"premium-w0-2d1f91-kserve-6965b558cf-59lgw\" (UID: \"fb84233d-25e0-48bc-8899-c3a3cb3cfeb9\") " pod="e2e-models-e2e-worker-w0-2d1f91/premium-w0-2d1f91-kserve-6965b558cf-59lgw"
Oct 30 21:55:23.979798 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:55:23.979635    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb84233d-25e0-48bc-8899-c3a3cb3cfeb9-tls-certs\") pod \"premium-w0-2d1f91-kserve-6965b558cf-59lgw\" (UID: \"fb84233d-25e0-48bc-8899-c3a3cb3cfeb9\") " pod="e2e-models-e2e-worker-w0-2d1f91/premium-w0-2d1f91-kserve-6965b558cf-59lgw"
Oct 30 21:55:23.979798 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:55:23.979675    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb84233d-25e0-48bc-8899-c3a3cb3cfeb9-model-cache\") pod \"premium-w0-2d1f91-kserve-6965b558cf-59lgw\" (UID: \"fb84233d-25e0-48bc-8899-c3a3cb3cfeb9\") " pod="e2e-models-e2e-worker-w0-2d1f91/premium-w0-2d1f91-kserve-6965b558cf-59lgw"
Oct 30 21:55:23.979798 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:55:23.979719    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8z9h2\" (UniqueName: \"kubernetes.io/projected/fb84233d-25e0-48bc-8899-c3a3cb3cfeb9-kube-api-access-8z9h2\") pod \"premium-w0-2d1f91-kserve-6965b558cf-59lgw\" (UID: \"fb84233d-25e0-48bc-8899-c3a3cb3cfeb9\") " pod="e2e-models-e2e-worker-w0-2d1f91/premium-w0-2d1f91-kserve-6965b558cf-59lgw"
Oct 30 21:55:23.979798 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:55:23.979739    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb84233d-25e0-48bc-8899-c3a3cb3cfeb9-tmp-dir\") pod \"premium-w0-2d1f91-kserve-6965b558cf-59lgw\" (UID: \"fb84233d-25e0-48bc-8899-c3a3cb3cfeb9\") " pod="e2e-models-e2e-worker-w0-2d1f91/premium-w0-2d1f91-kserve-6965b558cf-59lgw"
Oct 30 21:55:23.980027 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:55:23.979839    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb84233d-25e0-48bc-8899-c3a3cb3cfeb9-dshm\") pod \"premium-w0-2d1f91-kserve-6965b558cf-59lgw\" (UID: \"fb84233d-25e0-48bc-8899-c3a3cb3cfeb9\") " pod="e2e-models-e2e-worker-w0-2d1f91/premium-w0-2d1f91-kserve-6965b558cf-59lgw"
Oct 30 21:55:23.980027 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:55:23.980015    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb84233d-25e0-48bc-8899-c3a3cb3cfeb9-home\") pod \"premium-w0-2d1f91-kserve-6965b558cf-59lgw\" (UID: \"fb84233d-25e0-48bc-8899-c3a3cb3cfeb9\") " pod="e2e-models-e2e-worker-w0-2d1f91/premium-w0-2d1f91-kserve-6965b558cf-59lgw"
Oct 30 21:55:23.980133 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:55:23.980082    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb84233d-25e0-48bc-8899-c3a3cb3cfeb9-model-cache\") pod \"premium-w0-2d1f91-kserve-6965b558cf-59lgw\" (UID: \"fb84233d-25e0-48bc-8899-c3a3cb3cfeb9\") " pod="e2e-models-e2e-worker-w0-2d1f91/premium-w0-2d1f91-kserve-6965b558cf-59lgw"
Oct 30 21:55:23.980184 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:55:23.980146    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb84233d-25e0-48bc-8899-c3a3cb3cfeb9-tmp-dir\") pod \"premium-w0-2d1f91-kserve-6965b558cf-59lgw\" (UID: \"fb84233d-25e0-48bc-8899-c3a3cb3cfeb9\") " pod="e2e-models-e2e-worker-w0-2d1f91/premium-w0-2d1f91-kserve-6965b558cf-59lgw"
Oct 30 21:55:23.982019 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:55:23.981997    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb84233d-25e0-48bc-8899-c3a3cb3cfeb9-dshm\") pod \"premium-w0-2d1f91-kserve-6965b558cf-59lgw\" (UID: \"fb84233d-25e0-48bc-8899-c3a3cb3cfeb9\") " pod="e2e-models-e2e-worker-w0-2d1f91/premium-w0-2d1f91-kserve-6965b558cf-59lgw"
Oct 30 21:55:23.982260 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:55:23.982243    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb84233d-25e0-48bc-8899-c3a3cb3cfeb9-tls-certs\") pod \"premium-w0-2d1f91-kserve-6965b558cf-59lgw\" (UID: \"fb84233d-25e0-48bc-8899-c3a3cb3cfeb9\") " pod="e2e-models-e2e-worker-w0-2d1f91/premium-w0-2d1f91-kserve-6965b558cf-59lgw"
Oct 30 21:55:24.036811 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:55:24.036761    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8z9h2\" (UniqueName: \"kubernetes.io/projected/fb84233d-25e0-48bc-8899-c3a3cb3cfeb9-kube-api-access-8z9h2\") pod \"premium-w0-2d1f91-kserve-6965b558cf-59lgw\" (UID: \"fb84233d-25e0-48bc-8899-c3a3cb3cfeb9\") " pod="e2e-models-e2e-worker-w0-2d1f91/premium-w0-2d1f91-kserve-6965b558cf-59lgw"
Oct 30 21:55:24.099385 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:55:24.099356    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2d1f91/premium-w0-2d1f91-kserve-6965b558cf-59lgw"
Oct 30 21:55:24.494446 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:55:24.494415    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-2d1f91/premium-w0-2d1f91-kserve-6965b558cf-59lgw"]
Oct 30 21:55:24.498950 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:55:24.498912    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfb84233d_25e0_48bc_8899_c3a3cb3cfeb9.slice/crio-11885bf7aecadd2497da239c388e9f46d1c23c5a61f7b906ab4d432ba6db17ca WatchSource:0}: Error finding container 11885bf7aecadd2497da239c388e9f46d1c23c5a61f7b906ab4d432ba6db17ca: Status 404 returned error can't find the container with id 11885bf7aecadd2497da239c388e9f46d1c23c5a61f7b906ab4d432ba6db17ca
Oct 30 21:55:24.667303 ip-10-0-130-60 kubenswrapper[5061]: E1030 21:55:24.667268    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 21:55:24.789116 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:55:24.789032    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2d1f91/premium-w0-2d1f91-kserve-6965b558cf-59lgw" event={"ID":"fb84233d-25e0-48bc-8899-c3a3cb3cfeb9","Type":"ContainerStarted","Data":"161ec0d5ec48158bd3d0eac0af1fc29538a01f4d1682c350cb8e36c822315f22"}
Oct 30 21:55:24.789116 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:55:24.789081    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2d1f91/premium-w0-2d1f91-kserve-6965b558cf-59lgw" event={"ID":"fb84233d-25e0-48bc-8899-c3a3cb3cfeb9","Type":"ContainerStarted","Data":"11885bf7aecadd2497da239c388e9f46d1c23c5a61f7b906ab4d432ba6db17ca"}
Oct 30 21:55:24.902426 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:55:24.902376    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-2d1f91/premium-w0-2d1f91-kserve-6965b558cf-59lgw" podStartSLOduration=1.902362983 podStartE2EDuration="1.902362983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:55:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:55:24.89948992 +0000 UTC m=+778.512259697" watchObservedRunningTime="2026-10-30 21:55:24.902362983 +0000 UTC m=+778.515132735"
Oct 30 21:55:25.099802 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:55:25.099771    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2d1f91/premium-w0-2d1f91-kserve-6965b558cf-59lgw"
Oct 30 21:55:29.980214 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:55:29.980162    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-17d809/simulator-w4-17d809-kserve-78d67b6f4-7hsvb"
Oct 30 21:55:29.992784 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:55:29.992756    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-17d809/simulator-w4-17d809-kserve-78d67b6f4-7hsvb"
Oct 30 21:55:30.839588 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:55:30.839521    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-17d809/simulator-w4-17d809-kserve-78d67b6f4-7hsvb"
Oct 30 21:55:34.100477 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:55:34.100435    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-2d1f91/premium-w0-2d1f91-kserve-6965b558cf-59lgw"
Oct 30 21:55:34.112916 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:55:34.112891    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-2d1f91/premium-w0-2d1f91-kserve-6965b558cf-59lgw"
Oct 30 21:55:34.863244 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:55:34.863208    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2d1f91/premium-w0-2d1f91-kserve-6965b558cf-59lgw"
Oct 30 21:55:54.667271 ip-10-0-130-60 kubenswrapper[5061]: E1030 21:55:54.667232    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 21:56:02.234484 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:02.234444    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-14af5c/premium-w6-14af5c-kserve-6549647f4-f4gwg"]
Oct 30 21:56:02.238938 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:02.238914    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-14af5c/premium-w6-14af5c-kserve-6549647f4-f4gwg"
Oct 30 21:56:02.301010 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:02.300970    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2p6cm\" (UniqueName: \"kubernetes.io/projected/9b42ee01-c60e-462c-a0e0-b04a52014db6-kube-api-access-2p6cm\") pod \"premium-w6-14af5c-kserve-6549647f4-f4gwg\" (UID: \"9b42ee01-c60e-462c-a0e0-b04a52014db6\") " pod="e2e-models-e2e-worker-w6-14af5c/premium-w6-14af5c-kserve-6549647f4-f4gwg"
Oct 30 21:56:02.301200 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:02.301027    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b42ee01-c60e-462c-a0e0-b04a52014db6-home\") pod \"premium-w6-14af5c-kserve-6549647f4-f4gwg\" (UID: \"9b42ee01-c60e-462c-a0e0-b04a52014db6\") " pod="e2e-models-e2e-worker-w6-14af5c/premium-w6-14af5c-kserve-6549647f4-f4gwg"
Oct 30 21:56:02.301266 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:02.301218    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b42ee01-c60e-462c-a0e0-b04a52014db6-model-cache\") pod \"premium-w6-14af5c-kserve-6549647f4-f4gwg\" (UID: \"9b42ee01-c60e-462c-a0e0-b04a52014db6\") " pod="e2e-models-e2e-worker-w6-14af5c/premium-w6-14af5c-kserve-6549647f4-f4gwg"
Oct 30 21:56:02.301266 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:02.301261    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b42ee01-c60e-462c-a0e0-b04a52014db6-tls-certs\") pod \"premium-w6-14af5c-kserve-6549647f4-f4gwg\" (UID: \"9b42ee01-c60e-462c-a0e0-b04a52014db6\") " pod="e2e-models-e2e-worker-w6-14af5c/premium-w6-14af5c-kserve-6549647f4-f4gwg"
Oct 30 21:56:02.301372 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:02.301303    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b42ee01-c60e-462c-a0e0-b04a52014db6-tmp-dir\") pod \"premium-w6-14af5c-kserve-6549647f4-f4gwg\" (UID: \"9b42ee01-c60e-462c-a0e0-b04a52014db6\") " pod="e2e-models-e2e-worker-w6-14af5c/premium-w6-14af5c-kserve-6549647f4-f4gwg"
Oct 30 21:56:02.301372 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:02.301334    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b42ee01-c60e-462c-a0e0-b04a52014db6-dshm\") pod \"premium-w6-14af5c-kserve-6549647f4-f4gwg\" (UID: \"9b42ee01-c60e-462c-a0e0-b04a52014db6\") " pod="e2e-models-e2e-worker-w6-14af5c/premium-w6-14af5c-kserve-6549647f4-f4gwg"
Oct 30 21:56:02.336650 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:02.336615    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-14af5c\"/\"premium-w6-14af5c-kserve-self-signed-certs\""
Oct 30 21:56:02.339219 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:02.339193    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-14af5c/premium-w6-14af5c-kserve-6549647f4-f4gwg"]
Oct 30 21:56:02.402673 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:02.402636    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b42ee01-c60e-462c-a0e0-b04a52014db6-model-cache\") pod \"premium-w6-14af5c-kserve-6549647f4-f4gwg\" (UID: \"9b42ee01-c60e-462c-a0e0-b04a52014db6\") " pod="e2e-models-e2e-worker-w6-14af5c/premium-w6-14af5c-kserve-6549647f4-f4gwg"
Oct 30 21:56:02.402954 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:02.402707    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b42ee01-c60e-462c-a0e0-b04a52014db6-tls-certs\") pod \"premium-w6-14af5c-kserve-6549647f4-f4gwg\" (UID: \"9b42ee01-c60e-462c-a0e0-b04a52014db6\") " pod="e2e-models-e2e-worker-w6-14af5c/premium-w6-14af5c-kserve-6549647f4-f4gwg"
Oct 30 21:56:02.402954 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:02.402752    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b42ee01-c60e-462c-a0e0-b04a52014db6-tmp-dir\") pod \"premium-w6-14af5c-kserve-6549647f4-f4gwg\" (UID: \"9b42ee01-c60e-462c-a0e0-b04a52014db6\") " pod="e2e-models-e2e-worker-w6-14af5c/premium-w6-14af5c-kserve-6549647f4-f4gwg"
Oct 30 21:56:02.402954 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:02.402778    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b42ee01-c60e-462c-a0e0-b04a52014db6-dshm\") pod \"premium-w6-14af5c-kserve-6549647f4-f4gwg\" (UID: \"9b42ee01-c60e-462c-a0e0-b04a52014db6\") " pod="e2e-models-e2e-worker-w6-14af5c/premium-w6-14af5c-kserve-6549647f4-f4gwg"
Oct 30 21:56:02.402954 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:02.402807    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2p6cm\" (UniqueName: \"kubernetes.io/projected/9b42ee01-c60e-462c-a0e0-b04a52014db6-kube-api-access-2p6cm\") pod \"premium-w6-14af5c-kserve-6549647f4-f4gwg\" (UID: \"9b42ee01-c60e-462c-a0e0-b04a52014db6\") " pod="e2e-models-e2e-worker-w6-14af5c/premium-w6-14af5c-kserve-6549647f4-f4gwg"
Oct 30 21:56:02.402954 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:02.402843    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b42ee01-c60e-462c-a0e0-b04a52014db6-home\") pod \"premium-w6-14af5c-kserve-6549647f4-f4gwg\" (UID: \"9b42ee01-c60e-462c-a0e0-b04a52014db6\") " pod="e2e-models-e2e-worker-w6-14af5c/premium-w6-14af5c-kserve-6549647f4-f4gwg"
Oct 30 21:56:02.403238 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:02.403186    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b42ee01-c60e-462c-a0e0-b04a52014db6-model-cache\") pod \"premium-w6-14af5c-kserve-6549647f4-f4gwg\" (UID: \"9b42ee01-c60e-462c-a0e0-b04a52014db6\") " pod="e2e-models-e2e-worker-w6-14af5c/premium-w6-14af5c-kserve-6549647f4-f4gwg"
Oct 30 21:56:02.403295 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:02.403246    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b42ee01-c60e-462c-a0e0-b04a52014db6-home\") pod \"premium-w6-14af5c-kserve-6549647f4-f4gwg\" (UID: \"9b42ee01-c60e-462c-a0e0-b04a52014db6\") " pod="e2e-models-e2e-worker-w6-14af5c/premium-w6-14af5c-kserve-6549647f4-f4gwg"
Oct 30 21:56:02.403435 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:02.403410    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b42ee01-c60e-462c-a0e0-b04a52014db6-tmp-dir\") pod \"premium-w6-14af5c-kserve-6549647f4-f4gwg\" (UID: \"9b42ee01-c60e-462c-a0e0-b04a52014db6\") " pod="e2e-models-e2e-worker-w6-14af5c/premium-w6-14af5c-kserve-6549647f4-f4gwg"
Oct 30 21:56:02.405826 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:02.405803    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b42ee01-c60e-462c-a0e0-b04a52014db6-dshm\") pod \"premium-w6-14af5c-kserve-6549647f4-f4gwg\" (UID: \"9b42ee01-c60e-462c-a0e0-b04a52014db6\") " pod="e2e-models-e2e-worker-w6-14af5c/premium-w6-14af5c-kserve-6549647f4-f4gwg"
Oct 30 21:56:02.406266 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:02.406242    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b42ee01-c60e-462c-a0e0-b04a52014db6-tls-certs\") pod \"premium-w6-14af5c-kserve-6549647f4-f4gwg\" (UID: \"9b42ee01-c60e-462c-a0e0-b04a52014db6\") " pod="e2e-models-e2e-worker-w6-14af5c/premium-w6-14af5c-kserve-6549647f4-f4gwg"
Oct 30 21:56:02.461944 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:02.461913    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2p6cm\" (UniqueName: \"kubernetes.io/projected/9b42ee01-c60e-462c-a0e0-b04a52014db6-kube-api-access-2p6cm\") pod \"premium-w6-14af5c-kserve-6549647f4-f4gwg\" (UID: \"9b42ee01-c60e-462c-a0e0-b04a52014db6\") " pod="e2e-models-e2e-worker-w6-14af5c/premium-w6-14af5c-kserve-6549647f4-f4gwg"
Oct 30 21:56:02.550810 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:02.550776    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-14af5c/premium-w6-14af5c-kserve-6549647f4-f4gwg"
Oct 30 21:56:02.937404 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:02.937321    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-14af5c/premium-w6-14af5c-kserve-6549647f4-f4gwg"]
Oct 30 21:56:02.942466 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:56:02.942435    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9b42ee01_c60e_462c_a0e0_b04a52014db6.slice/crio-f9ac53dd94110da765c6e0a7aa3285b0028e172b5b934486cb1ffe0a1fd9a2b7 WatchSource:0}: Error finding container f9ac53dd94110da765c6e0a7aa3285b0028e172b5b934486cb1ffe0a1fd9a2b7: Status 404 returned error can't find the container with id f9ac53dd94110da765c6e0a7aa3285b0028e172b5b934486cb1ffe0a1fd9a2b7
Oct 30 21:56:03.027568 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:03.027513    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-14af5c/premium-w6-14af5c-kserve-6549647f4-f4gwg" event={"ID":"9b42ee01-c60e-462c-a0e0-b04a52014db6","Type":"ContainerStarted","Data":"b1b2212c3c7180a7832363d0cc2347107f1744364b4e0b1561dd4a8c2fd722cf"}
Oct 30 21:56:03.027727 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:03.027582    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-14af5c/premium-w6-14af5c-kserve-6549647f4-f4gwg" event={"ID":"9b42ee01-c60e-462c-a0e0-b04a52014db6","Type":"ContainerStarted","Data":"f9ac53dd94110da765c6e0a7aa3285b0028e172b5b934486cb1ffe0a1fd9a2b7"}
Oct 30 21:56:03.147166 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:03.147107    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-14af5c/premium-w6-14af5c-kserve-6549647f4-f4gwg" podStartSLOduration=1.147091627 podStartE2EDuration="1.147091627s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:56:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:56:03.144169229 +0000 UTC m=+816.756938986" watchObservedRunningTime="2026-10-30 21:56:03.147091627 +0000 UTC m=+816.759861442"
Oct 30 21:56:03.551622 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:03.551589    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-14af5c/premium-w6-14af5c-kserve-6549647f4-f4gwg"
Oct 30 21:56:12.551180 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:12.551146    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-14af5c/premium-w6-14af5c-kserve-6549647f4-f4gwg"
Oct 30 21:56:12.563523 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:12.563495    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-14af5c/premium-w6-14af5c-kserve-6549647f4-f4gwg"
Oct 30 21:56:13.095309 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:13.095282    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-14af5c/premium-w6-14af5c-kserve-6549647f4-f4gwg"
Oct 30 21:56:24.667131 ip-10-0-130-60 kubenswrapper[5061]: E1030 21:56:24.667093    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 21:56:34.443004 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:34.442972    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-5be0d1/premium-w2-5be0d1-kserve-759b6fb995-c6gp4"]
Oct 30 21:56:34.447577 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:34.447556    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-5be0d1/premium-w2-5be0d1-kserve-759b6fb995-c6gp4"
Oct 30 21:56:34.491883 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:34.491843    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grw7m\" (UniqueName: \"kubernetes.io/projected/933f2470-68de-4768-a05b-6c454f16f5f2-kube-api-access-grw7m\") pod \"premium-w2-5be0d1-kserve-759b6fb995-c6gp4\" (UID: \"933f2470-68de-4768-a05b-6c454f16f5f2\") " pod="e2e-models-e2e-worker-w2-5be0d1/premium-w2-5be0d1-kserve-759b6fb995-c6gp4"
Oct 30 21:56:34.492075 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:34.491902    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/933f2470-68de-4768-a05b-6c454f16f5f2-dshm\") pod \"premium-w2-5be0d1-kserve-759b6fb995-c6gp4\" (UID: \"933f2470-68de-4768-a05b-6c454f16f5f2\") " pod="e2e-models-e2e-worker-w2-5be0d1/premium-w2-5be0d1-kserve-759b6fb995-c6gp4"
Oct 30 21:56:34.492255 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:34.492218    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/933f2470-68de-4768-a05b-6c454f16f5f2-home\") pod \"premium-w2-5be0d1-kserve-759b6fb995-c6gp4\" (UID: \"933f2470-68de-4768-a05b-6c454f16f5f2\") " pod="e2e-models-e2e-worker-w2-5be0d1/premium-w2-5be0d1-kserve-759b6fb995-c6gp4"
Oct 30 21:56:34.492379 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:34.492342    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/933f2470-68de-4768-a05b-6c454f16f5f2-tls-certs\") pod \"premium-w2-5be0d1-kserve-759b6fb995-c6gp4\" (UID: \"933f2470-68de-4768-a05b-6c454f16f5f2\") " pod="e2e-models-e2e-worker-w2-5be0d1/premium-w2-5be0d1-kserve-759b6fb995-c6gp4"
Oct 30 21:56:34.492379 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:34.492370    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/933f2470-68de-4768-a05b-6c454f16f5f2-model-cache\") pod \"premium-w2-5be0d1-kserve-759b6fb995-c6gp4\" (UID: \"933f2470-68de-4768-a05b-6c454f16f5f2\") " pod="e2e-models-e2e-worker-w2-5be0d1/premium-w2-5be0d1-kserve-759b6fb995-c6gp4"
Oct 30 21:56:34.492482 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:34.492416    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/933f2470-68de-4768-a05b-6c454f16f5f2-tmp-dir\") pod \"premium-w2-5be0d1-kserve-759b6fb995-c6gp4\" (UID: \"933f2470-68de-4768-a05b-6c454f16f5f2\") " pod="e2e-models-e2e-worker-w2-5be0d1/premium-w2-5be0d1-kserve-759b6fb995-c6gp4"
Oct 30 21:56:34.498099 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:34.498072    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-5be0d1\"/\"openshift-service-ca.crt\""
Oct 30 21:56:34.545063 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:34.545030    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-5be0d1\"/\"kube-root-ca.crt\""
Oct 30 21:56:34.545234 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:34.545196    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-5be0d1\"/\"default-dockercfg-mkbfj\""
Oct 30 21:56:34.545394 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:34.545376    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-5be0d1\"/\"premium-w2-5be0d1-kserve-self-signed-certs\""
Oct 30 21:56:34.549863 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:34.549835    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-5be0d1/premium-w2-5be0d1-kserve-759b6fb995-c6gp4"]
Oct 30 21:56:34.597012 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:34.596968    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/933f2470-68de-4768-a05b-6c454f16f5f2-tls-certs\") pod \"premium-w2-5be0d1-kserve-759b6fb995-c6gp4\" (UID: \"933f2470-68de-4768-a05b-6c454f16f5f2\") " pod="e2e-models-e2e-worker-w2-5be0d1/premium-w2-5be0d1-kserve-759b6fb995-c6gp4"
Oct 30 21:56:34.597278 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:34.597027    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/933f2470-68de-4768-a05b-6c454f16f5f2-model-cache\") pod \"premium-w2-5be0d1-kserve-759b6fb995-c6gp4\" (UID: \"933f2470-68de-4768-a05b-6c454f16f5f2\") " pod="e2e-models-e2e-worker-w2-5be0d1/premium-w2-5be0d1-kserve-759b6fb995-c6gp4"
Oct 30 21:56:34.597278 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:34.597086    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/933f2470-68de-4768-a05b-6c454f16f5f2-tmp-dir\") pod \"premium-w2-5be0d1-kserve-759b6fb995-c6gp4\" (UID: \"933f2470-68de-4768-a05b-6c454f16f5f2\") " pod="e2e-models-e2e-worker-w2-5be0d1/premium-w2-5be0d1-kserve-759b6fb995-c6gp4"
Oct 30 21:56:34.597278 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:34.597199    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grw7m\" (UniqueName: \"kubernetes.io/projected/933f2470-68de-4768-a05b-6c454f16f5f2-kube-api-access-grw7m\") pod \"premium-w2-5be0d1-kserve-759b6fb995-c6gp4\" (UID: \"933f2470-68de-4768-a05b-6c454f16f5f2\") " pod="e2e-models-e2e-worker-w2-5be0d1/premium-w2-5be0d1-kserve-759b6fb995-c6gp4"
Oct 30 21:56:34.597278 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:34.597243    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/933f2470-68de-4768-a05b-6c454f16f5f2-dshm\") pod \"premium-w2-5be0d1-kserve-759b6fb995-c6gp4\" (UID: \"933f2470-68de-4768-a05b-6c454f16f5f2\") " pod="e2e-models-e2e-worker-w2-5be0d1/premium-w2-5be0d1-kserve-759b6fb995-c6gp4"
Oct 30 21:56:34.597675 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:34.597300    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/933f2470-68de-4768-a05b-6c454f16f5f2-home\") pod \"premium-w2-5be0d1-kserve-759b6fb995-c6gp4\" (UID: \"933f2470-68de-4768-a05b-6c454f16f5f2\") " pod="e2e-models-e2e-worker-w2-5be0d1/premium-w2-5be0d1-kserve-759b6fb995-c6gp4"
Oct 30 21:56:34.600247 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:34.599783    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/933f2470-68de-4768-a05b-6c454f16f5f2-home\") pod \"premium-w2-5be0d1-kserve-759b6fb995-c6gp4\" (UID: \"933f2470-68de-4768-a05b-6c454f16f5f2\") " pod="e2e-models-e2e-worker-w2-5be0d1/premium-w2-5be0d1-kserve-759b6fb995-c6gp4"
Oct 30 21:56:34.603837 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:34.603809    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/933f2470-68de-4768-a05b-6c454f16f5f2-tmp-dir\") pod \"premium-w2-5be0d1-kserve-759b6fb995-c6gp4\" (UID: \"933f2470-68de-4768-a05b-6c454f16f5f2\") " pod="e2e-models-e2e-worker-w2-5be0d1/premium-w2-5be0d1-kserve-759b6fb995-c6gp4"
Oct 30 21:56:34.608188 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:34.608155    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/933f2470-68de-4768-a05b-6c454f16f5f2-model-cache\") pod \"premium-w2-5be0d1-kserve-759b6fb995-c6gp4\" (UID: \"933f2470-68de-4768-a05b-6c454f16f5f2\") " pod="e2e-models-e2e-worker-w2-5be0d1/premium-w2-5be0d1-kserve-759b6fb995-c6gp4"
Oct 30 21:56:34.612403 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:34.612377    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/933f2470-68de-4768-a05b-6c454f16f5f2-tls-certs\") pod \"premium-w2-5be0d1-kserve-759b6fb995-c6gp4\" (UID: \"933f2470-68de-4768-a05b-6c454f16f5f2\") " pod="e2e-models-e2e-worker-w2-5be0d1/premium-w2-5be0d1-kserve-759b6fb995-c6gp4"
Oct 30 21:56:34.614819 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:34.614750    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/933f2470-68de-4768-a05b-6c454f16f5f2-dshm\") pod \"premium-w2-5be0d1-kserve-759b6fb995-c6gp4\" (UID: \"933f2470-68de-4768-a05b-6c454f16f5f2\") " pod="e2e-models-e2e-worker-w2-5be0d1/premium-w2-5be0d1-kserve-759b6fb995-c6gp4"
Oct 30 21:56:34.663317 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:34.663289    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grw7m\" (UniqueName: \"kubernetes.io/projected/933f2470-68de-4768-a05b-6c454f16f5f2-kube-api-access-grw7m\") pod \"premium-w2-5be0d1-kserve-759b6fb995-c6gp4\" (UID: \"933f2470-68de-4768-a05b-6c454f16f5f2\") " pod="e2e-models-e2e-worker-w2-5be0d1/premium-w2-5be0d1-kserve-759b6fb995-c6gp4"
Oct 30 21:56:34.805550 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:34.805161    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-5be0d1/premium-w2-5be0d1-kserve-759b6fb995-c6gp4"
Oct 30 21:56:35.192736 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:35.192651    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-5be0d1/premium-w2-5be0d1-kserve-759b6fb995-c6gp4"]
Oct 30 21:56:35.197921 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:56:35.197888    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod933f2470_68de_4768_a05b_6c454f16f5f2.slice/crio-29e316f76da4cd7ec06b6c1c82720dd9f32b432d9df4776b8225fcb281dc8eee WatchSource:0}: Error finding container 29e316f76da4cd7ec06b6c1c82720dd9f32b432d9df4776b8225fcb281dc8eee: Status 404 returned error can't find the container with id 29e316f76da4cd7ec06b6c1c82720dd9f32b432d9df4776b8225fcb281dc8eee
Oct 30 21:56:35.217969 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:35.217926    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-5be0d1/premium-w2-5be0d1-kserve-759b6fb995-c6gp4" event={"ID":"933f2470-68de-4768-a05b-6c454f16f5f2","Type":"ContainerStarted","Data":"29e316f76da4cd7ec06b6c1c82720dd9f32b432d9df4776b8225fcb281dc8eee"}
Oct 30 21:56:36.226193 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:36.226145    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-5be0d1/premium-w2-5be0d1-kserve-759b6fb995-c6gp4" event={"ID":"933f2470-68de-4768-a05b-6c454f16f5f2","Type":"ContainerStarted","Data":"f511e810551d2e435d6289dbdbb835de89b6cf797170c3f5615ae6351f28459c"}
Oct 30 21:56:36.335866 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:36.335802    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-5be0d1/premium-w2-5be0d1-kserve-759b6fb995-c6gp4" podStartSLOduration=2.3357817020000002 podStartE2EDuration="2.335781702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:56:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:56:36.334868204 +0000 UTC m=+849.947637972" watchObservedRunningTime="2026-10-30 21:56:36.335781702 +0000 UTC m=+849.948551461"
Oct 30 21:56:36.806202 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:36.806167    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-5be0d1/premium-w2-5be0d1-kserve-759b6fb995-c6gp4"
Oct 30 21:56:39.291061 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:39.290982    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-17d809/premium-w4-17d809-kserve-7c9bdbbcbc-5mtmm"]
Oct 30 21:56:39.295201 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:39.295180    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-17d809/premium-w4-17d809-kserve-7c9bdbbcbc-5mtmm"
Oct 30 21:56:39.393199 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:39.393165    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-17d809\"/\"premium-w4-17d809-kserve-self-signed-certs\""
Oct 30 21:56:39.395932 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:39.395910    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-17d809/premium-w4-17d809-kserve-7c9bdbbcbc-5mtmm"]
Oct 30 21:56:39.398756 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:39.398736    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e0eca81-7db1-482b-8dd6-5b28fb89e010-tls-certs\") pod \"premium-w4-17d809-kserve-7c9bdbbcbc-5mtmm\" (UID: \"1e0eca81-7db1-482b-8dd6-5b28fb89e010\") " pod="e2e-models-e2e-worker-w4-17d809/premium-w4-17d809-kserve-7c9bdbbcbc-5mtmm"
Oct 30 21:56:39.398830 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:39.398800    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e0eca81-7db1-482b-8dd6-5b28fb89e010-tmp-dir\") pod \"premium-w4-17d809-kserve-7c9bdbbcbc-5mtmm\" (UID: \"1e0eca81-7db1-482b-8dd6-5b28fb89e010\") " pod="e2e-models-e2e-worker-w4-17d809/premium-w4-17d809-kserve-7c9bdbbcbc-5mtmm"
Oct 30 21:56:39.398900 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:39.398878    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e0eca81-7db1-482b-8dd6-5b28fb89e010-model-cache\") pod \"premium-w4-17d809-kserve-7c9bdbbcbc-5mtmm\" (UID: \"1e0eca81-7db1-482b-8dd6-5b28fb89e010\") " pod="e2e-models-e2e-worker-w4-17d809/premium-w4-17d809-kserve-7c9bdbbcbc-5mtmm"
Oct 30 21:56:39.398953 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:39.398932    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhqwr\" (UniqueName: \"kubernetes.io/projected/1e0eca81-7db1-482b-8dd6-5b28fb89e010-kube-api-access-vhqwr\") pod \"premium-w4-17d809-kserve-7c9bdbbcbc-5mtmm\" (UID: \"1e0eca81-7db1-482b-8dd6-5b28fb89e010\") " pod="e2e-models-e2e-worker-w4-17d809/premium-w4-17d809-kserve-7c9bdbbcbc-5mtmm"
Oct 30 21:56:39.399051 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:39.399034    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e0eca81-7db1-482b-8dd6-5b28fb89e010-home\") pod \"premium-w4-17d809-kserve-7c9bdbbcbc-5mtmm\" (UID: \"1e0eca81-7db1-482b-8dd6-5b28fb89e010\") " pod="e2e-models-e2e-worker-w4-17d809/premium-w4-17d809-kserve-7c9bdbbcbc-5mtmm"
Oct 30 21:56:39.399104 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:39.399069    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e0eca81-7db1-482b-8dd6-5b28fb89e010-dshm\") pod \"premium-w4-17d809-kserve-7c9bdbbcbc-5mtmm\" (UID: \"1e0eca81-7db1-482b-8dd6-5b28fb89e010\") " pod="e2e-models-e2e-worker-w4-17d809/premium-w4-17d809-kserve-7c9bdbbcbc-5mtmm"
Oct 30 21:56:39.500188 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:39.500154    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e0eca81-7db1-482b-8dd6-5b28fb89e010-home\") pod \"premium-w4-17d809-kserve-7c9bdbbcbc-5mtmm\" (UID: \"1e0eca81-7db1-482b-8dd6-5b28fb89e010\") " pod="e2e-models-e2e-worker-w4-17d809/premium-w4-17d809-kserve-7c9bdbbcbc-5mtmm"
Oct 30 21:56:39.500372 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:39.500202    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e0eca81-7db1-482b-8dd6-5b28fb89e010-dshm\") pod \"premium-w4-17d809-kserve-7c9bdbbcbc-5mtmm\" (UID: \"1e0eca81-7db1-482b-8dd6-5b28fb89e010\") " pod="e2e-models-e2e-worker-w4-17d809/premium-w4-17d809-kserve-7c9bdbbcbc-5mtmm"
Oct 30 21:56:39.500372 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:39.500248    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e0eca81-7db1-482b-8dd6-5b28fb89e010-tls-certs\") pod \"premium-w4-17d809-kserve-7c9bdbbcbc-5mtmm\" (UID: \"1e0eca81-7db1-482b-8dd6-5b28fb89e010\") " pod="e2e-models-e2e-worker-w4-17d809/premium-w4-17d809-kserve-7c9bdbbcbc-5mtmm"
Oct 30 21:56:39.500456 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:39.500407    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e0eca81-7db1-482b-8dd6-5b28fb89e010-tmp-dir\") pod \"premium-w4-17d809-kserve-7c9bdbbcbc-5mtmm\" (UID: \"1e0eca81-7db1-482b-8dd6-5b28fb89e010\") " pod="e2e-models-e2e-worker-w4-17d809/premium-w4-17d809-kserve-7c9bdbbcbc-5mtmm"
Oct 30 21:56:39.500506 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:39.500467    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e0eca81-7db1-482b-8dd6-5b28fb89e010-model-cache\") pod \"premium-w4-17d809-kserve-7c9bdbbcbc-5mtmm\" (UID: \"1e0eca81-7db1-482b-8dd6-5b28fb89e010\") " pod="e2e-models-e2e-worker-w4-17d809/premium-w4-17d809-kserve-7c9bdbbcbc-5mtmm"
Oct 30 21:56:39.500599 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:39.500554    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhqwr\" (UniqueName: \"kubernetes.io/projected/1e0eca81-7db1-482b-8dd6-5b28fb89e010-kube-api-access-vhqwr\") pod \"premium-w4-17d809-kserve-7c9bdbbcbc-5mtmm\" (UID: \"1e0eca81-7db1-482b-8dd6-5b28fb89e010\") " pod="e2e-models-e2e-worker-w4-17d809/premium-w4-17d809-kserve-7c9bdbbcbc-5mtmm"
Oct 30 21:56:39.500668 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:39.500609    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e0eca81-7db1-482b-8dd6-5b28fb89e010-home\") pod \"premium-w4-17d809-kserve-7c9bdbbcbc-5mtmm\" (UID: \"1e0eca81-7db1-482b-8dd6-5b28fb89e010\") " pod="e2e-models-e2e-worker-w4-17d809/premium-w4-17d809-kserve-7c9bdbbcbc-5mtmm"
Oct 30 21:56:39.500802 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:39.500780    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e0eca81-7db1-482b-8dd6-5b28fb89e010-tmp-dir\") pod \"premium-w4-17d809-kserve-7c9bdbbcbc-5mtmm\" (UID: \"1e0eca81-7db1-482b-8dd6-5b28fb89e010\") " pod="e2e-models-e2e-worker-w4-17d809/premium-w4-17d809-kserve-7c9bdbbcbc-5mtmm"
Oct 30 21:56:39.500874 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:39.500808    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e0eca81-7db1-482b-8dd6-5b28fb89e010-model-cache\") pod \"premium-w4-17d809-kserve-7c9bdbbcbc-5mtmm\" (UID: \"1e0eca81-7db1-482b-8dd6-5b28fb89e010\") " pod="e2e-models-e2e-worker-w4-17d809/premium-w4-17d809-kserve-7c9bdbbcbc-5mtmm"
Oct 30 21:56:39.502571 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:39.502518    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e0eca81-7db1-482b-8dd6-5b28fb89e010-dshm\") pod \"premium-w4-17d809-kserve-7c9bdbbcbc-5mtmm\" (UID: \"1e0eca81-7db1-482b-8dd6-5b28fb89e010\") " pod="e2e-models-e2e-worker-w4-17d809/premium-w4-17d809-kserve-7c9bdbbcbc-5mtmm"
Oct 30 21:56:39.502851 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:39.502834    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e0eca81-7db1-482b-8dd6-5b28fb89e010-tls-certs\") pod \"premium-w4-17d809-kserve-7c9bdbbcbc-5mtmm\" (UID: \"1e0eca81-7db1-482b-8dd6-5b28fb89e010\") " pod="e2e-models-e2e-worker-w4-17d809/premium-w4-17d809-kserve-7c9bdbbcbc-5mtmm"
Oct 30 21:56:39.556336 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:39.556304    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhqwr\" (UniqueName: \"kubernetes.io/projected/1e0eca81-7db1-482b-8dd6-5b28fb89e010-kube-api-access-vhqwr\") pod \"premium-w4-17d809-kserve-7c9bdbbcbc-5mtmm\" (UID: \"1e0eca81-7db1-482b-8dd6-5b28fb89e010\") " pod="e2e-models-e2e-worker-w4-17d809/premium-w4-17d809-kserve-7c9bdbbcbc-5mtmm"
Oct 30 21:56:39.607344 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:39.607309    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-17d809/premium-w4-17d809-kserve-7c9bdbbcbc-5mtmm"
Oct 30 21:56:39.999252 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:39.999165    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-17d809/premium-w4-17d809-kserve-7c9bdbbcbc-5mtmm"]
Oct 30 21:56:40.004487 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:56:40.004457    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1e0eca81_7db1_482b_8dd6_5b28fb89e010.slice/crio-aca4ef922fa5d6485f175c0d37068ce891fb8c5983cb682152ff7e72ceb8e246 WatchSource:0}: Error finding container aca4ef922fa5d6485f175c0d37068ce891fb8c5983cb682152ff7e72ceb8e246: Status 404 returned error can't find the container with id aca4ef922fa5d6485f175c0d37068ce891fb8c5983cb682152ff7e72ceb8e246
Oct 30 21:56:40.252033 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:40.251944    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-17d809/premium-w4-17d809-kserve-7c9bdbbcbc-5mtmm" event={"ID":"1e0eca81-7db1-482b-8dd6-5b28fb89e010","Type":"ContainerStarted","Data":"81f3a64544cb556ee2539afd8521fc7525f84b418f74c42af6074580ef77e5f7"}
Oct 30 21:56:40.252033 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:40.251989    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-17d809/premium-w4-17d809-kserve-7c9bdbbcbc-5mtmm" event={"ID":"1e0eca81-7db1-482b-8dd6-5b28fb89e010","Type":"ContainerStarted","Data":"aca4ef922fa5d6485f175c0d37068ce891fb8c5983cb682152ff7e72ceb8e246"}
Oct 30 21:56:40.364196 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:40.364145    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-17d809/premium-w4-17d809-kserve-7c9bdbbcbc-5mtmm" podStartSLOduration=1.364132505 podStartE2EDuration="1.364132505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:56:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:56:40.363001138 +0000 UTC m=+853.975770896" watchObservedRunningTime="2026-10-30 21:56:40.364132505 +0000 UTC m=+853.976902262"
Oct 30 21:56:40.607586 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:40.607556    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-17d809/premium-w4-17d809-kserve-7c9bdbbcbc-5mtmm"
Oct 30 21:56:44.806715 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:44.806681    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-5be0d1/premium-w2-5be0d1-kserve-759b6fb995-c6gp4"
Oct 30 21:56:44.824731 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:44.824706    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-5be0d1/premium-w2-5be0d1-kserve-759b6fb995-c6gp4"
Oct 30 21:56:45.291305 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:45.291222    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-5be0d1/premium-w2-5be0d1-kserve-759b6fb995-c6gp4"
Oct 30 21:56:49.607920 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:49.607887    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-17d809/premium-w4-17d809-kserve-7c9bdbbcbc-5mtmm"
Oct 30 21:56:49.620737 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:49.620713    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-17d809/premium-w4-17d809-kserve-7c9bdbbcbc-5mtmm"
Oct 30 21:56:50.329134 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:50.329107    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-17d809/premium-w4-17d809-kserve-7c9bdbbcbc-5mtmm"
Oct 30 21:56:51.191943 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:51.191910    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-shwf8"]
Oct 30 21:56:51.196381 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:51.196358    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-shwf8"
Oct 30 21:56:51.240142 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:51.240106    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f137aa3-0e26-4526-92b0-2636c979075f-istio-envoy\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-shwf8\" (UID: \"9f137aa3-0e26-4526-92b0-2636c979075f\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-shwf8"
Oct 30 21:56:51.240330 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:51.240157    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsxp6\" (UniqueName: \"kubernetes.io/projected/9f137aa3-0e26-4526-92b0-2636c979075f-kube-api-access-jsxp6\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-shwf8\" (UID: \"9f137aa3-0e26-4526-92b0-2636c979075f\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-shwf8"
Oct 30 21:56:51.240330 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:51.240236    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f137aa3-0e26-4526-92b0-2636c979075f-istiod-ca-cert\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-shwf8\" (UID: \"9f137aa3-0e26-4526-92b0-2636c979075f\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-shwf8"
Oct 30 21:56:51.240330 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:51.240302    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f137aa3-0e26-4526-92b0-2636c979075f-workload-certs\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-shwf8\" (UID: \"9f137aa3-0e26-4526-92b0-2636c979075f\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-shwf8"
Oct 30 21:56:51.240515 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:51.240358    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f137aa3-0e26-4526-92b0-2636c979075f-istio-token\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-shwf8\" (UID: \"9f137aa3-0e26-4526-92b0-2636c979075f\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-shwf8"
Oct 30 21:56:51.240515 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:51.240389    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f137aa3-0e26-4526-92b0-2636c979075f-workload-socket\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-shwf8\" (UID: \"9f137aa3-0e26-4526-92b0-2636c979075f\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-shwf8"
Oct 30 21:56:51.240676 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:51.240559    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f137aa3-0e26-4526-92b0-2636c979075f-credential-socket\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-shwf8\" (UID: \"9f137aa3-0e26-4526-92b0-2636c979075f\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-shwf8"
Oct 30 21:56:51.240676 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:51.240607    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f137aa3-0e26-4526-92b0-2636c979075f-istio-podinfo\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-shwf8\" (UID: \"9f137aa3-0e26-4526-92b0-2636c979075f\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-shwf8"
Oct 30 21:56:51.240787 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:51.240694    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f137aa3-0e26-4526-92b0-2636c979075f-istio-data\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-shwf8\" (UID: \"9f137aa3-0e26-4526-92b0-2636c979075f\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-shwf8"
Oct 30 21:56:51.246077 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:51.246052    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-14af5c-openshift-default-dockercfg-f5mqp\""
Oct 30 21:56:51.298038 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:51.298006    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-shwf8"]
Oct 30 21:56:51.341867 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:51.341839    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f137aa3-0e26-4526-92b0-2636c979075f-istio-token\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-shwf8\" (UID: \"9f137aa3-0e26-4526-92b0-2636c979075f\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-shwf8"
Oct 30 21:56:51.342048 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:51.341875    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f137aa3-0e26-4526-92b0-2636c979075f-workload-socket\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-shwf8\" (UID: \"9f137aa3-0e26-4526-92b0-2636c979075f\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-shwf8"
Oct 30 21:56:51.342048 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:51.341897    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f137aa3-0e26-4526-92b0-2636c979075f-credential-socket\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-shwf8\" (UID: \"9f137aa3-0e26-4526-92b0-2636c979075f\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-shwf8"
Oct 30 21:56:51.342048 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:51.341912    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f137aa3-0e26-4526-92b0-2636c979075f-istio-podinfo\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-shwf8\" (UID: \"9f137aa3-0e26-4526-92b0-2636c979075f\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-shwf8"
Oct 30 21:56:51.342166 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:51.342104    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f137aa3-0e26-4526-92b0-2636c979075f-istio-data\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-shwf8\" (UID: \"9f137aa3-0e26-4526-92b0-2636c979075f\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-shwf8"
Oct 30 21:56:51.342253 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:51.342217    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f137aa3-0e26-4526-92b0-2636c979075f-istio-envoy\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-shwf8\" (UID: \"9f137aa3-0e26-4526-92b0-2636c979075f\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-shwf8"
Oct 30 21:56:51.342327 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:51.342260    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsxp6\" (UniqueName: \"kubernetes.io/projected/9f137aa3-0e26-4526-92b0-2636c979075f-kube-api-access-jsxp6\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-shwf8\" (UID: \"9f137aa3-0e26-4526-92b0-2636c979075f\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-shwf8"
Oct 30 21:56:51.342381 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:51.342343    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f137aa3-0e26-4526-92b0-2636c979075f-istiod-ca-cert\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-shwf8\" (UID: \"9f137aa3-0e26-4526-92b0-2636c979075f\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-shwf8"
Oct 30 21:56:51.342436 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:51.342402    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f137aa3-0e26-4526-92b0-2636c979075f-workload-socket\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-shwf8\" (UID: \"9f137aa3-0e26-4526-92b0-2636c979075f\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-shwf8"
Oct 30 21:56:51.342436 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:51.342408    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f137aa3-0e26-4526-92b0-2636c979075f-workload-certs\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-shwf8\" (UID: \"9f137aa3-0e26-4526-92b0-2636c979075f\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-shwf8"
Oct 30 21:56:51.342571 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:51.342442    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f137aa3-0e26-4526-92b0-2636c979075f-credential-socket\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-shwf8\" (UID: \"9f137aa3-0e26-4526-92b0-2636c979075f\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-shwf8"
Oct 30 21:56:51.342571 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:51.342474    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f137aa3-0e26-4526-92b0-2636c979075f-istio-data\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-shwf8\" (UID: \"9f137aa3-0e26-4526-92b0-2636c979075f\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-shwf8"
Oct 30 21:56:51.342778 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:51.342758    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f137aa3-0e26-4526-92b0-2636c979075f-workload-certs\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-shwf8\" (UID: \"9f137aa3-0e26-4526-92b0-2636c979075f\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-shwf8"
Oct 30 21:56:51.343062 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:51.343038    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f137aa3-0e26-4526-92b0-2636c979075f-istiod-ca-cert\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-shwf8\" (UID: \"9f137aa3-0e26-4526-92b0-2636c979075f\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-shwf8"
Oct 30 21:56:51.344571 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:51.344516    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f137aa3-0e26-4526-92b0-2636c979075f-istio-envoy\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-shwf8\" (UID: \"9f137aa3-0e26-4526-92b0-2636c979075f\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-shwf8"
Oct 30 21:56:51.344656 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:51.344584    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f137aa3-0e26-4526-92b0-2636c979075f-istio-podinfo\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-shwf8\" (UID: \"9f137aa3-0e26-4526-92b0-2636c979075f\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-shwf8"
Oct 30 21:56:51.397461 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:51.397433    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f137aa3-0e26-4526-92b0-2636c979075f-istio-token\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-shwf8\" (UID: \"9f137aa3-0e26-4526-92b0-2636c979075f\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-shwf8"
Oct 30 21:56:51.397685 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:51.397662    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsxp6\" (UniqueName: \"kubernetes.io/projected/9f137aa3-0e26-4526-92b0-2636c979075f-kube-api-access-jsxp6\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-shwf8\" (UID: \"9f137aa3-0e26-4526-92b0-2636c979075f\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-shwf8"
Oct 30 21:56:51.511699 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:51.511613    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-shwf8"
Oct 30 21:56:51.941048 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:51.940792    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-shwf8"]
Oct 30 21:56:51.945809 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:56:51.945782    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f137aa3_0e26_4526_92b0_2636c979075f.slice/crio-829e67ec674c7f44931ff9675f91091236f1f795da8f5999295d1fb04bcb1ac6 WatchSource:0}: Error finding container 829e67ec674c7f44931ff9675f91091236f1f795da8f5999295d1fb04bcb1ac6: Status 404 returned error can't find the container with id 829e67ec674c7f44931ff9675f91091236f1f795da8f5999295d1fb04bcb1ac6
Oct 30 21:56:51.948250 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:51.948218    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:56:51.948334 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:51.948284    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:56:51.948334 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:51.948331    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:56:52.342276 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:52.342201    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-shwf8" event={"ID":"9f137aa3-0e26-4526-92b0-2636c979075f","Type":"ContainerStarted","Data":"c85d056c85a8b6e789e406e2e1db65cd780b651b9239857a721b511f348c1b76"}
Oct 30 21:56:52.342829 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:52.342344    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-shwf8" event={"ID":"9f137aa3-0e26-4526-92b0-2636c979075f","Type":"ContainerStarted","Data":"829e67ec674c7f44931ff9675f91091236f1f795da8f5999295d1fb04bcb1ac6"}
Oct 30 21:56:52.511917 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:52.511884    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-shwf8"
Oct 30 21:56:52.523086 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:52.523028    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-shwf8" podStartSLOduration=1.52301491 podStartE2EDuration="1.52301491s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:56:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:56:52.521880487 +0000 UTC m=+866.134650256" watchObservedRunningTime="2026-10-30 21:56:52.52301491 +0000 UTC m=+866.135784669"
Oct 30 21:56:53.527551 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:53.525462    5061 patch_prober.go:29] interesting pod/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-shwf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Oct 30 21:56:53.527551 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:53.525518    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-shwf8" podUID="9f137aa3-0e26-4526-92b0-2636c979075f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Oct 30 21:56:54.524551 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:54.512836    5061 patch_prober.go:29] interesting pod/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-shwf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Oct 30 21:56:54.524551 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:54.512903    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-shwf8" podUID="9f137aa3-0e26-4526-92b0-2636c979075f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Oct 30 21:56:54.666257 ip-10-0-130-60 kubenswrapper[5061]: E1030 21:56:54.666212    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 21:56:55.523551 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:55.519866    5061 patch_prober.go:29] interesting pod/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-shwf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Oct 30 21:56:55.523551 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:55.519937    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-shwf8" podUID="9f137aa3-0e26-4526-92b0-2636c979075f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Oct 30 21:56:56.516842 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:56.516808    5061 patch_prober.go:29] interesting pod/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-shwf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Oct 30 21:56:56.517414 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:56.517383    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-shwf8" podUID="9f137aa3-0e26-4526-92b0-2636c979075f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Oct 30 21:56:57.528550 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:57.527623    5061 patch_prober.go:29] interesting pod/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-shwf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Oct 30 21:56:57.528550 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:57.527692    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-shwf8" podUID="9f137aa3-0e26-4526-92b0-2636c979075f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Oct 30 21:56:58.512148 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:58.512113    5061 patch_prober.go:29] interesting pod/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-shwf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Oct 30 21:56:58.512322 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:58.512177    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-shwf8" podUID="9f137aa3-0e26-4526-92b0-2636c979075f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Oct 30 21:56:59.516696 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:59.516666    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-shwf8"
Oct 30 21:56:59.517172 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:59.517060    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-shwf8"
Oct 30 21:56:59.517734 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:56:59.517714    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-shwf8"
Oct 30 21:57:05.303677 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:05.303641    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-km4g4"]
Oct 30 21:57:05.304255 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:05.303842    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-km4g4" podUID="cb4b4d8d-f667-4816-b351-e147ddc48224" containerName="istio-proxy" containerID="cri-o://607adf5c76fe05b67177a64fdd761144fa7b3f8d6f98cff10e60a2127b1bd0a5" gracePeriod=30
Oct 30 21:57:05.351002 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:05.350971    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-58dhn"]
Oct 30 21:57:05.351195 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:05.351159    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-58dhn" podUID="99439df0-9c24-4a03-9dff-e95af6f3f3bd" containerName="istio-proxy" containerID="cri-o://a76a1f744b3dc65ae379f968708b9b3d2731dc2331f9249f74dd50d8cc36989c" gracePeriod=30
Oct 30 21:57:05.353612 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:05.353565    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-94lvp"]
Oct 30 21:57:05.353790 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:05.353764    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-94lvp" podUID="62dfda76-164c-4a47-9328-2468648266fd" containerName="istio-proxy" containerID="cri-o://2016103dfa49cc28b3ea3fe995371d78fde903346c574a6487057308b6db36de" gracePeriod=30
Oct 30 21:57:05.397894 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:05.397868    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vcstl"]
Oct 30 21:57:05.398068 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:05.398037    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vcstl" podUID="877aedb6-e77a-4cfa-9120-2650f6e39604" containerName="istio-proxy" containerID="cri-o://8336bb2d481b832628f08f76872ebddde9e71962d6ac9c654192c10e1eb437e1" gracePeriod=30
Oct 30 21:57:09.046341 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:09.046299    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-94lvp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 21:57:09.046728 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:09.046371    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-94lvp" podUID="62dfda76-164c-4a47-9328-2468648266fd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 21:57:10.270059 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:10.270022    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-58dhn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 21:57:10.270450 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:10.270082    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-58dhn" podUID="99439df0-9c24-4a03-9dff-e95af6f3f3bd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 21:57:10.481391 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:10.481360    5061 generic.go:356] "Generic (PLEG): container finished" podID="62dfda76-164c-4a47-9328-2468648266fd" containerID="2016103dfa49cc28b3ea3fe995371d78fde903346c574a6487057308b6db36de" exitCode=0
Oct 30 21:57:10.481561 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:10.481439    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-94lvp" event={"ID":"62dfda76-164c-4a47-9328-2468648266fd","Type":"ContainerDied","Data":"2016103dfa49cc28b3ea3fe995371d78fde903346c574a6487057308b6db36de"}
Oct 30 21:57:10.483519 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:10.483499    5061 generic.go:356] "Generic (PLEG): container finished" podID="99439df0-9c24-4a03-9dff-e95af6f3f3bd" containerID="a76a1f744b3dc65ae379f968708b9b3d2731dc2331f9249f74dd50d8cc36989c" exitCode=0
Oct 30 21:57:10.483632 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:10.483593    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-58dhn" event={"ID":"99439df0-9c24-4a03-9dff-e95af6f3f3bd","Type":"ContainerDied","Data":"a76a1f744b3dc65ae379f968708b9b3d2731dc2331f9249f74dd50d8cc36989c"}
Oct 30 21:57:10.483693 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:10.483671    5061 scope.go:122] "RemoveContainer" containerID="40106f33775b1b36f0a7625562f0efa8af2bc9d98b0fe0f31067b2e8c776b69e"
Oct 30 21:57:10.485187 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:10.485166    5061 generic.go:356] "Generic (PLEG): container finished" podID="cb4b4d8d-f667-4816-b351-e147ddc48224" containerID="607adf5c76fe05b67177a64fdd761144fa7b3f8d6f98cff10e60a2127b1bd0a5" exitCode=0
Oct 30 21:57:10.485281 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:10.485247    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-km4g4" event={"ID":"cb4b4d8d-f667-4816-b351-e147ddc48224","Type":"ContainerDied","Data":"607adf5c76fe05b67177a64fdd761144fa7b3f8d6f98cff10e60a2127b1bd0a5"}
Oct 30 21:57:10.662706 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:10.662608    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-km4g4"
Oct 30 21:57:10.701711 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:10.701681    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-58dhn"
Oct 30 21:57:10.711708 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:10.710738    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb4b4d8d-f667-4816-b351-e147ddc48224-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb4b4d8d-f667-4816-b351-e147ddc48224-istio-data\") pod \"cb4b4d8d-f667-4816-b351-e147ddc48224\" (UID: \"cb4b4d8d-f667-4816-b351-e147ddc48224\") "
Oct 30 21:57:10.711708 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:10.710815    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb4b4d8d-f667-4816-b351-e147ddc48224-istio-token\" (UniqueName: \"kubernetes.io/projected/cb4b4d8d-f667-4816-b351-e147ddc48224-istio-token\") pod \"cb4b4d8d-f667-4816-b351-e147ddc48224\" (UID: \"cb4b4d8d-f667-4816-b351-e147ddc48224\") "
Oct 30 21:57:10.711708 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:10.710876    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb4b4d8d-f667-4816-b351-e147ddc48224-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb4b4d8d-f667-4816-b351-e147ddc48224-istiod-ca-cert\") pod \"cb4b4d8d-f667-4816-b351-e147ddc48224\" (UID: \"cb4b4d8d-f667-4816-b351-e147ddc48224\") "
Oct 30 21:57:10.711708 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:10.710903    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb4b4d8d-f667-4816-b351-e147ddc48224-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb4b4d8d-f667-4816-b351-e147ddc48224-workload-socket\") pod \"cb4b4d8d-f667-4816-b351-e147ddc48224\" (UID: \"cb4b4d8d-f667-4816-b351-e147ddc48224\") "
Oct 30 21:57:10.711708 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:10.710937    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb4b4d8d-f667-4816-b351-e147ddc48224-kube-api-access-6zsv9\" (UniqueName: \"kubernetes.io/projected/cb4b4d8d-f667-4816-b351-e147ddc48224-kube-api-access-6zsv9\") pod \"cb4b4d8d-f667-4816-b351-e147ddc48224\" (UID: \"cb4b4d8d-f667-4816-b351-e147ddc48224\") "
Oct 30 21:57:10.711708 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:10.710981    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb4b4d8d-f667-4816-b351-e147ddc48224-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb4b4d8d-f667-4816-b351-e147ddc48224-workload-certs\") pod \"cb4b4d8d-f667-4816-b351-e147ddc48224\" (UID: \"cb4b4d8d-f667-4816-b351-e147ddc48224\") "
Oct 30 21:57:10.711708 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:10.711033    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb4b4d8d-f667-4816-b351-e147ddc48224-istio-data" pod "cb4b4d8d-f667-4816-b351-e147ddc48224" (UID: "cb4b4d8d-f667-4816-b351-e147ddc48224"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:57:10.711708 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:10.711057    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb4b4d8d-f667-4816-b351-e147ddc48224-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb4b4d8d-f667-4816-b351-e147ddc48224-credential-socket\") pod \"cb4b4d8d-f667-4816-b351-e147ddc48224\" (UID: \"cb4b4d8d-f667-4816-b351-e147ddc48224\") "
Oct 30 21:57:10.711708 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:10.711088    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb4b4d8d-f667-4816-b351-e147ddc48224-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb4b4d8d-f667-4816-b351-e147ddc48224-istio-envoy\") pod \"cb4b4d8d-f667-4816-b351-e147ddc48224\" (UID: \"cb4b4d8d-f667-4816-b351-e147ddc48224\") "
Oct 30 21:57:10.711708 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:10.711260    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb4b4d8d-f667-4816-b351-e147ddc48224-workload-socket" pod "cb4b4d8d-f667-4816-b351-e147ddc48224" (UID: "cb4b4d8d-f667-4816-b351-e147ddc48224"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:57:10.711708 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:10.711277    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cb4b4d8d-f667-4816-b351-e147ddc48224-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb4b4d8d-f667-4816-b351-e147ddc48224-istio-podinfo\") pod \"cb4b4d8d-f667-4816-b351-e147ddc48224\" (UID: \"cb4b4d8d-f667-4816-b351-e147ddc48224\") "
Oct 30 21:57:10.711708 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:10.711629    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb4b4d8d-f667-4816-b351-e147ddc48224-credential-socket" pod "cb4b4d8d-f667-4816-b351-e147ddc48224" (UID: "cb4b4d8d-f667-4816-b351-e147ddc48224"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:57:10.712492 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:10.711994    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb4b4d8d-f667-4816-b351-e147ddc48224-workload-certs" pod "cb4b4d8d-f667-4816-b351-e147ddc48224" (UID: "cb4b4d8d-f667-4816-b351-e147ddc48224"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:57:10.712492 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:10.712011    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb4b4d8d-f667-4816-b351-e147ddc48224-istiod-ca-cert" pod "cb4b4d8d-f667-4816-b351-e147ddc48224" (UID: "cb4b4d8d-f667-4816-b351-e147ddc48224"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:57:10.712783 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:10.712760    5061 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb4b4d8d-f667-4816-b351-e147ddc48224-istio-data\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:10.712857 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:10.712789    5061 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb4b4d8d-f667-4816-b351-e147ddc48224-istiod-ca-cert\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:10.712857 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:10.712805    5061 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb4b4d8d-f667-4816-b351-e147ddc48224-workload-socket\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:10.712857 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:10.712819    5061 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb4b4d8d-f667-4816-b351-e147ddc48224-workload-certs\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:10.712857 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:10.712833    5061 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb4b4d8d-f667-4816-b351-e147ddc48224-credential-socket\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:10.714852 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:10.714826    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb4b4d8d-f667-4816-b351-e147ddc48224-istio-token" pod "cb4b4d8d-f667-4816-b351-e147ddc48224" (UID: "cb4b4d8d-f667-4816-b351-e147ddc48224"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:57:10.725593 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:10.723637    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb4b4d8d-f667-4816-b351-e147ddc48224-istio-envoy" pod "cb4b4d8d-f667-4816-b351-e147ddc48224" (UID: "cb4b4d8d-f667-4816-b351-e147ddc48224"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:57:10.725593 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:10.723652    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cb4b4d8d-f667-4816-b351-e147ddc48224-istio-podinfo" pod "cb4b4d8d-f667-4816-b351-e147ddc48224" (UID: "cb4b4d8d-f667-4816-b351-e147ddc48224"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:57:10.725593 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:10.723689    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb4b4d8d-f667-4816-b351-e147ddc48224-kube-api-access-6zsv9" pod "cb4b4d8d-f667-4816-b351-e147ddc48224" (UID: "cb4b4d8d-f667-4816-b351-e147ddc48224"). InnerVolumeSpecName "kube-api-access-6zsv9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:57:10.763138 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:10.763118    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-94lvp"
Oct 30 21:57:10.765771 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:10.765750    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vcstl"
Oct 30 21:57:10.813677 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:10.813642    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62dfda76-164c-4a47-9328-2468648266fd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62dfda76-164c-4a47-9328-2468648266fd-credential-socket\") pod \"62dfda76-164c-4a47-9328-2468648266fd\" (UID: \"62dfda76-164c-4a47-9328-2468648266fd\") "
Oct 30 21:57:10.813854 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:10.813717    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99439df0-9c24-4a03-9dff-e95af6f3f3bd-istio-token\" (UniqueName: \"kubernetes.io/projected/99439df0-9c24-4a03-9dff-e95af6f3f3bd-istio-token\") pod \"99439df0-9c24-4a03-9dff-e95af6f3f3bd\" (UID: \"99439df0-9c24-4a03-9dff-e95af6f3f3bd\") "
Oct 30 21:57:10.813854 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:10.813753    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/877aedb6-e77a-4cfa-9120-2650f6e39604-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/877aedb6-e77a-4cfa-9120-2650f6e39604-istiod-ca-cert\") pod \"877aedb6-e77a-4cfa-9120-2650f6e39604\" (UID: \"877aedb6-e77a-4cfa-9120-2650f6e39604\") "
Oct 30 21:57:10.813854 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:10.813786    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62dfda76-164c-4a47-9328-2468648266fd-istio-token\" (UniqueName: \"kubernetes.io/projected/62dfda76-164c-4a47-9328-2468648266fd-istio-token\") pod \"62dfda76-164c-4a47-9328-2468648266fd\" (UID: \"62dfda76-164c-4a47-9328-2468648266fd\") "
Oct 30 21:57:10.813854 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:10.813833    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99439df0-9c24-4a03-9dff-e95af6f3f3bd-kube-api-access-qsgst\" (UniqueName: \"kubernetes.io/projected/99439df0-9c24-4a03-9dff-e95af6f3f3bd-kube-api-access-qsgst\") pod \"99439df0-9c24-4a03-9dff-e95af6f3f3bd\" (UID: \"99439df0-9c24-4a03-9dff-e95af6f3f3bd\") "
Oct 30 21:57:10.814086 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:10.813868    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/877aedb6-e77a-4cfa-9120-2650f6e39604-istio-token\" (UniqueName: \"kubernetes.io/projected/877aedb6-e77a-4cfa-9120-2650f6e39604-istio-token\") pod \"877aedb6-e77a-4cfa-9120-2650f6e39604\" (UID: \"877aedb6-e77a-4cfa-9120-2650f6e39604\") "
Oct 30 21:57:10.814086 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:10.813928    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/877aedb6-e77a-4cfa-9120-2650f6e39604-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/877aedb6-e77a-4cfa-9120-2650f6e39604-workload-socket\") pod \"877aedb6-e77a-4cfa-9120-2650f6e39604\" (UID: \"877aedb6-e77a-4cfa-9120-2650f6e39604\") "
Oct 30 21:57:10.814086 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:10.813944    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62dfda76-164c-4a47-9328-2468648266fd-credential-socket" pod "62dfda76-164c-4a47-9328-2468648266fd" (UID: "62dfda76-164c-4a47-9328-2468648266fd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:57:10.814086 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:10.813959    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99439df0-9c24-4a03-9dff-e95af6f3f3bd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99439df0-9c24-4a03-9dff-e95af6f3f3bd-istiod-ca-cert\") pod \"99439df0-9c24-4a03-9dff-e95af6f3f3bd\" (UID: \"99439df0-9c24-4a03-9dff-e95af6f3f3bd\") "
Oct 30 21:57:10.814086 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:10.814002    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62dfda76-164c-4a47-9328-2468648266fd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62dfda76-164c-4a47-9328-2468648266fd-istio-data\") pod \"62dfda76-164c-4a47-9328-2468648266fd\" (UID: \"62dfda76-164c-4a47-9328-2468648266fd\") "
Oct 30 21:57:10.814086 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:10.814031    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62dfda76-164c-4a47-9328-2468648266fd-kube-api-access-r7xrd\" (UniqueName: \"kubernetes.io/projected/62dfda76-164c-4a47-9328-2468648266fd-kube-api-access-r7xrd\") pod \"62dfda76-164c-4a47-9328-2468648266fd\" (UID: \"62dfda76-164c-4a47-9328-2468648266fd\") "
Oct 30 21:57:10.814086 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:10.814075    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62dfda76-164c-4a47-9328-2468648266fd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62dfda76-164c-4a47-9328-2468648266fd-istiod-ca-cert\") pod \"62dfda76-164c-4a47-9328-2468648266fd\" (UID: \"62dfda76-164c-4a47-9328-2468648266fd\") "
Oct 30 21:57:10.814457 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:10.814117    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/99439df0-9c24-4a03-9dff-e95af6f3f3bd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99439df0-9c24-4a03-9dff-e95af6f3f3bd-istio-podinfo\") pod \"99439df0-9c24-4a03-9dff-e95af6f3f3bd\" (UID: \"99439df0-9c24-4a03-9dff-e95af6f3f3bd\") "
Oct 30 21:57:10.814457 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:10.814174    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/877aedb6-e77a-4cfa-9120-2650f6e39604-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/877aedb6-e77a-4cfa-9120-2650f6e39604-workload-certs\") pod \"877aedb6-e77a-4cfa-9120-2650f6e39604\" (UID: \"877aedb6-e77a-4cfa-9120-2650f6e39604\") "
Oct 30 21:57:10.814457 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:10.814216    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/877aedb6-e77a-4cfa-9120-2650f6e39604-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/877aedb6-e77a-4cfa-9120-2650f6e39604-istio-podinfo\") pod \"877aedb6-e77a-4cfa-9120-2650f6e39604\" (UID: \"877aedb6-e77a-4cfa-9120-2650f6e39604\") "
Oct 30 21:57:10.814457 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:10.814262    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62dfda76-164c-4a47-9328-2468648266fd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62dfda76-164c-4a47-9328-2468648266fd-workload-certs\") pod \"62dfda76-164c-4a47-9328-2468648266fd\" (UID: \"62dfda76-164c-4a47-9328-2468648266fd\") "
Oct 30 21:57:10.814457 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:10.814314    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/877aedb6-e77a-4cfa-9120-2650f6e39604-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/877aedb6-e77a-4cfa-9120-2650f6e39604-istio-envoy\") pod \"877aedb6-e77a-4cfa-9120-2650f6e39604\" (UID: \"877aedb6-e77a-4cfa-9120-2650f6e39604\") "
Oct 30 21:57:10.814457 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:10.814340    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62dfda76-164c-4a47-9328-2468648266fd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62dfda76-164c-4a47-9328-2468648266fd-istio-envoy\") pod \"62dfda76-164c-4a47-9328-2468648266fd\" (UID: \"62dfda76-164c-4a47-9328-2468648266fd\") "
Oct 30 21:57:10.814457 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:10.814372    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/877aedb6-e77a-4cfa-9120-2650f6e39604-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/877aedb6-e77a-4cfa-9120-2650f6e39604-istio-data\") pod \"877aedb6-e77a-4cfa-9120-2650f6e39604\" (UID: \"877aedb6-e77a-4cfa-9120-2650f6e39604\") "
Oct 30 21:57:10.814457 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:10.814410    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62dfda76-164c-4a47-9328-2468648266fd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62dfda76-164c-4a47-9328-2468648266fd-workload-socket\") pod \"62dfda76-164c-4a47-9328-2468648266fd\" (UID: \"62dfda76-164c-4a47-9328-2468648266fd\") "
Oct 30 21:57:10.814457 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:10.814424    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62dfda76-164c-4a47-9328-2468648266fd-istio-data" pod "62dfda76-164c-4a47-9328-2468648266fd" (UID: "62dfda76-164c-4a47-9328-2468648266fd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:57:10.814996 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:10.814465    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/877aedb6-e77a-4cfa-9120-2650f6e39604-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/877aedb6-e77a-4cfa-9120-2650f6e39604-credential-socket\") pod \"877aedb6-e77a-4cfa-9120-2650f6e39604\" (UID: \"877aedb6-e77a-4cfa-9120-2650f6e39604\") "
Oct 30 21:57:10.814996 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:10.814504    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99439df0-9c24-4a03-9dff-e95af6f3f3bd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99439df0-9c24-4a03-9dff-e95af6f3f3bd-istio-data\") pod \"99439df0-9c24-4a03-9dff-e95af6f3f3bd\" (UID: \"99439df0-9c24-4a03-9dff-e95af6f3f3bd\") "
Oct 30 21:57:10.814996 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:10.814552    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/62dfda76-164c-4a47-9328-2468648266fd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62dfda76-164c-4a47-9328-2468648266fd-istio-podinfo\") pod \"62dfda76-164c-4a47-9328-2468648266fd\" (UID: \"62dfda76-164c-4a47-9328-2468648266fd\") "
Oct 30 21:57:10.814996 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:10.814589    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99439df0-9c24-4a03-9dff-e95af6f3f3bd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99439df0-9c24-4a03-9dff-e95af6f3f3bd-istio-envoy\") pod \"99439df0-9c24-4a03-9dff-e95af6f3f3bd\" (UID: \"99439df0-9c24-4a03-9dff-e95af6f3f3bd\") "
Oct 30 21:57:10.814996 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:10.814621    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/877aedb6-e77a-4cfa-9120-2650f6e39604-kube-api-access-42h2w\" (UniqueName: \"kubernetes.io/projected/877aedb6-e77a-4cfa-9120-2650f6e39604-kube-api-access-42h2w\") pod \"877aedb6-e77a-4cfa-9120-2650f6e39604\" (UID: \"877aedb6-e77a-4cfa-9120-2650f6e39604\") "
Oct 30 21:57:10.814996 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:10.814662    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99439df0-9c24-4a03-9dff-e95af6f3f3bd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99439df0-9c24-4a03-9dff-e95af6f3f3bd-credential-socket\") pod \"99439df0-9c24-4a03-9dff-e95af6f3f3bd\" (UID: \"99439df0-9c24-4a03-9dff-e95af6f3f3bd\") "
Oct 30 21:57:10.814996 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:10.814707    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99439df0-9c24-4a03-9dff-e95af6f3f3bd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99439df0-9c24-4a03-9dff-e95af6f3f3bd-workload-certs\") pod \"99439df0-9c24-4a03-9dff-e95af6f3f3bd\" (UID: \"99439df0-9c24-4a03-9dff-e95af6f3f3bd\") "
Oct 30 21:57:10.814996 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:10.814743    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99439df0-9c24-4a03-9dff-e95af6f3f3bd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99439df0-9c24-4a03-9dff-e95af6f3f3bd-workload-socket\") pod \"99439df0-9c24-4a03-9dff-e95af6f3f3bd\" (UID: \"99439df0-9c24-4a03-9dff-e95af6f3f3bd\") "
Oct 30 21:57:10.815432 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:10.815030    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62dfda76-164c-4a47-9328-2468648266fd-workload-socket" pod "62dfda76-164c-4a47-9328-2468648266fd" (UID: "62dfda76-164c-4a47-9328-2468648266fd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:57:10.815432 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:10.815231    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99439df0-9c24-4a03-9dff-e95af6f3f3bd-workload-socket" pod "99439df0-9c24-4a03-9dff-e95af6f3f3bd" (UID: "99439df0-9c24-4a03-9dff-e95af6f3f3bd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:57:10.815555 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:10.815430    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/877aedb6-e77a-4cfa-9120-2650f6e39604-credential-socket" pod "877aedb6-e77a-4cfa-9120-2650f6e39604" (UID: "877aedb6-e77a-4cfa-9120-2650f6e39604"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:57:10.815555 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:10.815472    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99439df0-9c24-4a03-9dff-e95af6f3f3bd-istiod-ca-cert" pod "99439df0-9c24-4a03-9dff-e95af6f3f3bd" (UID: "99439df0-9c24-4a03-9dff-e95af6f3f3bd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:57:10.815668 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:10.815556    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/877aedb6-e77a-4cfa-9120-2650f6e39604-workload-socket" pod "877aedb6-e77a-4cfa-9120-2650f6e39604" (UID: "877aedb6-e77a-4cfa-9120-2650f6e39604"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:57:10.815668 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:10.815655    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99439df0-9c24-4a03-9dff-e95af6f3f3bd-istio-data" pod "99439df0-9c24-4a03-9dff-e95af6f3f3bd" (UID: "99439df0-9c24-4a03-9dff-e95af6f3f3bd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:57:10.815883 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:10.815860    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62dfda76-164c-4a47-9328-2468648266fd-workload-certs" pod "62dfda76-164c-4a47-9328-2468648266fd" (UID: "62dfda76-164c-4a47-9328-2468648266fd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:57:10.816168 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:10.816147    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/877aedb6-e77a-4cfa-9120-2650f6e39604-istio-data" pod "877aedb6-e77a-4cfa-9120-2650f6e39604" (UID: "877aedb6-e77a-4cfa-9120-2650f6e39604"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:57:10.816891 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:10.816866    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/877aedb6-e77a-4cfa-9120-2650f6e39604-workload-certs" pod "877aedb6-e77a-4cfa-9120-2650f6e39604" (UID: "877aedb6-e77a-4cfa-9120-2650f6e39604"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:57:10.817025 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:10.816877    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62dfda76-164c-4a47-9328-2468648266fd-istiod-ca-cert" pod "62dfda76-164c-4a47-9328-2468648266fd" (UID: "62dfda76-164c-4a47-9328-2468648266fd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:57:10.817269 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:10.817248    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/877aedb6-e77a-4cfa-9120-2650f6e39604-istiod-ca-cert" pod "877aedb6-e77a-4cfa-9120-2650f6e39604" (UID: "877aedb6-e77a-4cfa-9120-2650f6e39604"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:57:10.817386 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:10.817287    5061 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb4b4d8d-f667-4816-b351-e147ddc48224-istio-token\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:10.817680 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:10.817661    5061 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62dfda76-164c-4a47-9328-2468648266fd-istio-data\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:10.817792 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:10.817779    5061 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62dfda76-164c-4a47-9328-2468648266fd-istiod-ca-cert\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:10.817886 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:10.817872    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zsv9\" (UniqueName: \"kubernetes.io/projected/cb4b4d8d-f667-4816-b351-e147ddc48224-kube-api-access-6zsv9\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:10.817977 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:10.817965    5061 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/877aedb6-e77a-4cfa-9120-2650f6e39604-workload-certs\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:10.818066 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:10.818053    5061 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62dfda76-164c-4a47-9328-2468648266fd-workload-certs\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:10.818153 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:10.818140    5061 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/877aedb6-e77a-4cfa-9120-2650f6e39604-istio-data\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:10.818247 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:10.818225    5061 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb4b4d8d-f667-4816-b351-e147ddc48224-istio-envoy\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:10.818348 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:10.818326    5061 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62dfda76-164c-4a47-9328-2468648266fd-workload-socket\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:10.818456 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:10.818350    5061 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/877aedb6-e77a-4cfa-9120-2650f6e39604-credential-socket\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:10.818456 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:10.818366    5061 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99439df0-9c24-4a03-9dff-e95af6f3f3bd-istio-data\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:10.818456 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:10.818382    5061 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99439df0-9c24-4a03-9dff-e95af6f3f3bd-workload-socket\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:10.818456 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:10.818396    5061 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62dfda76-164c-4a47-9328-2468648266fd-credential-socket\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:10.818456 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:10.818411    5061 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb4b4d8d-f667-4816-b351-e147ddc48224-istio-podinfo\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:10.818456 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:10.818424    5061 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/877aedb6-e77a-4cfa-9120-2650f6e39604-workload-socket\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:10.818456 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:10.818436    5061 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99439df0-9c24-4a03-9dff-e95af6f3f3bd-istiod-ca-cert\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:10.818456 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:10.817611    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99439df0-9c24-4a03-9dff-e95af6f3f3bd-credential-socket" pod "99439df0-9c24-4a03-9dff-e95af6f3f3bd" (UID: "99439df0-9c24-4a03-9dff-e95af6f3f3bd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:57:10.818918 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:10.818892    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99439df0-9c24-4a03-9dff-e95af6f3f3bd-workload-certs" pod "99439df0-9c24-4a03-9dff-e95af6f3f3bd" (UID: "99439df0-9c24-4a03-9dff-e95af6f3f3bd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:57:10.822204 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:10.821999    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/62dfda76-164c-4a47-9328-2468648266fd-istio-podinfo" pod "62dfda76-164c-4a47-9328-2468648266fd" (UID: "62dfda76-164c-4a47-9328-2468648266fd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:57:10.822204 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:10.822149    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/877aedb6-e77a-4cfa-9120-2650f6e39604-istio-token" pod "877aedb6-e77a-4cfa-9120-2650f6e39604" (UID: "877aedb6-e77a-4cfa-9120-2650f6e39604"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:57:10.822204 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:10.822177    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62dfda76-164c-4a47-9328-2468648266fd-kube-api-access-r7xrd" pod "62dfda76-164c-4a47-9328-2468648266fd" (UID: "62dfda76-164c-4a47-9328-2468648266fd"). InnerVolumeSpecName "kube-api-access-r7xrd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:57:10.822454 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:10.822252    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99439df0-9c24-4a03-9dff-e95af6f3f3bd-kube-api-access-qsgst" pod "99439df0-9c24-4a03-9dff-e95af6f3f3bd" (UID: "99439df0-9c24-4a03-9dff-e95af6f3f3bd"). InnerVolumeSpecName "kube-api-access-qsgst". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:57:10.824054 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:10.824027    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/877aedb6-e77a-4cfa-9120-2650f6e39604-istio-podinfo" pod "877aedb6-e77a-4cfa-9120-2650f6e39604" (UID: "877aedb6-e77a-4cfa-9120-2650f6e39604"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:57:10.824148 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:10.824043    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99439df0-9c24-4a03-9dff-e95af6f3f3bd-istio-envoy" pod "99439df0-9c24-4a03-9dff-e95af6f3f3bd" (UID: "99439df0-9c24-4a03-9dff-e95af6f3f3bd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:57:10.824148 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:10.824105    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62dfda76-164c-4a47-9328-2468648266fd-istio-envoy" pod "62dfda76-164c-4a47-9328-2468648266fd" (UID: "62dfda76-164c-4a47-9328-2468648266fd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:57:10.824233 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:10.824154    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99439df0-9c24-4a03-9dff-e95af6f3f3bd-istio-token" pod "99439df0-9c24-4a03-9dff-e95af6f3f3bd" (UID: "99439df0-9c24-4a03-9dff-e95af6f3f3bd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:57:10.824233 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:10.824177    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/877aedb6-e77a-4cfa-9120-2650f6e39604-kube-api-access-42h2w" pod "877aedb6-e77a-4cfa-9120-2650f6e39604" (UID: "877aedb6-e77a-4cfa-9120-2650f6e39604"). InnerVolumeSpecName "kube-api-access-42h2w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:57:10.824302 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:10.824276    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/877aedb6-e77a-4cfa-9120-2650f6e39604-istio-envoy" pod "877aedb6-e77a-4cfa-9120-2650f6e39604" (UID: "877aedb6-e77a-4cfa-9120-2650f6e39604"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:57:10.824554 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:10.824510    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62dfda76-164c-4a47-9328-2468648266fd-istio-token" pod "62dfda76-164c-4a47-9328-2468648266fd" (UID: "62dfda76-164c-4a47-9328-2468648266fd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:57:10.824822 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:10.824783    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/99439df0-9c24-4a03-9dff-e95af6f3f3bd-istio-podinfo" pod "99439df0-9c24-4a03-9dff-e95af6f3f3bd" (UID: "99439df0-9c24-4a03-9dff-e95af6f3f3bd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:57:10.919070 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:10.919036    5061 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99439df0-9c24-4a03-9dff-e95af6f3f3bd-istio-token\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:10.919070 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:10.919062    5061 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/877aedb6-e77a-4cfa-9120-2650f6e39604-istiod-ca-cert\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:10.919305 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:10.919087    5061 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62dfda76-164c-4a47-9328-2468648266fd-istio-token\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:10.919305 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:10.919110    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qsgst\" (UniqueName: \"kubernetes.io/projected/99439df0-9c24-4a03-9dff-e95af6f3f3bd-kube-api-access-qsgst\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:10.919305 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:10.919121    5061 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/877aedb6-e77a-4cfa-9120-2650f6e39604-istio-token\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:10.919305 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:10.919129    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7xrd\" (UniqueName: \"kubernetes.io/projected/62dfda76-164c-4a47-9328-2468648266fd-kube-api-access-r7xrd\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:10.919305 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:10.919137    5061 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99439df0-9c24-4a03-9dff-e95af6f3f3bd-istio-podinfo\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:10.919305 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:10.919147    5061 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/877aedb6-e77a-4cfa-9120-2650f6e39604-istio-podinfo\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:10.919305 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:10.919156    5061 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/877aedb6-e77a-4cfa-9120-2650f6e39604-istio-envoy\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:10.919305 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:10.919164    5061 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62dfda76-164c-4a47-9328-2468648266fd-istio-envoy\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:10.919305 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:10.919172    5061 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62dfda76-164c-4a47-9328-2468648266fd-istio-podinfo\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:10.919305 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:10.919180    5061 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99439df0-9c24-4a03-9dff-e95af6f3f3bd-istio-envoy\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:10.919305 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:10.919189    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42h2w\" (UniqueName: \"kubernetes.io/projected/877aedb6-e77a-4cfa-9120-2650f6e39604-kube-api-access-42h2w\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:10.919305 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:10.919197    5061 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99439df0-9c24-4a03-9dff-e95af6f3f3bd-credential-socket\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:10.919305 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:10.919205    5061 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99439df0-9c24-4a03-9dff-e95af6f3f3bd-workload-certs\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:11.492668 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:11.492636    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-94lvp"
Oct 30 21:57:11.493135 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:11.492635    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-94lvp" event={"ID":"62dfda76-164c-4a47-9328-2468648266fd","Type":"ContainerDied","Data":"71e182a04b195ab447238d66c602d8d9f4ca312a3880b8c50548031c0b06ce6e"}
Oct 30 21:57:11.493135 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:11.492782    5061 scope.go:122] "RemoveContainer" containerID="2016103dfa49cc28b3ea3fe995371d78fde903346c574a6487057308b6db36de"
Oct 30 21:57:11.494061 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:11.494026    5061 generic.go:356] "Generic (PLEG): container finished" podID="877aedb6-e77a-4cfa-9120-2650f6e39604" containerID="8336bb2d481b832628f08f76872ebddde9e71962d6ac9c654192c10e1eb437e1" exitCode=0
Oct 30 21:57:11.494146 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:11.494067    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vcstl"
Oct 30 21:57:11.494146 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:11.494093    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vcstl" event={"ID":"877aedb6-e77a-4cfa-9120-2650f6e39604","Type":"ContainerDied","Data":"8336bb2d481b832628f08f76872ebddde9e71962d6ac9c654192c10e1eb437e1"}
Oct 30 21:57:11.494146 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:11.494134    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vcstl" event={"ID":"877aedb6-e77a-4cfa-9120-2650f6e39604","Type":"ContainerDied","Data":"6ca341b7f170072dacf1c198443a6f92f35165b12dbf74e0a054d768094de698"}
Oct 30 21:57:11.495570 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:11.495547    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-58dhn"
Oct 30 21:57:11.495692 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:11.495551    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-58dhn" event={"ID":"99439df0-9c24-4a03-9dff-e95af6f3f3bd","Type":"ContainerDied","Data":"2e029d24647864ca6d551995ab09cccc000f777078ef06fd69fae01a779c45fa"}
Oct 30 21:57:11.496993 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:11.496973    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-km4g4"
Oct 30 21:57:11.497120 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:11.497091    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-km4g4" event={"ID":"cb4b4d8d-f667-4816-b351-e147ddc48224","Type":"ContainerDied","Data":"486b5bbb2637ba27922a72e3e74b6f137d56765123c443d2e1eca7b364794287"}
Oct 30 21:57:11.507756 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:11.507737    5061 scope.go:122] "RemoveContainer" containerID="8336bb2d481b832628f08f76872ebddde9e71962d6ac9c654192c10e1eb437e1"
Oct 30 21:57:11.516586 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:11.516568    5061 scope.go:122] "RemoveContainer" containerID="8336bb2d481b832628f08f76872ebddde9e71962d6ac9c654192c10e1eb437e1"
Oct 30 21:57:11.516831 ip-10-0-130-60 kubenswrapper[5061]: E1030 21:57:11.516813    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8336bb2d481b832628f08f76872ebddde9e71962d6ac9c654192c10e1eb437e1\": container with ID starting with 8336bb2d481b832628f08f76872ebddde9e71962d6ac9c654192c10e1eb437e1 not found: ID does not exist" containerID="8336bb2d481b832628f08f76872ebddde9e71962d6ac9c654192c10e1eb437e1"
Oct 30 21:57:11.516873 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:11.516840    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8336bb2d481b832628f08f76872ebddde9e71962d6ac9c654192c10e1eb437e1"} err="failed to get container status \"8336bb2d481b832628f08f76872ebddde9e71962d6ac9c654192c10e1eb437e1\": rpc error: code = NotFound desc = could not find container \"8336bb2d481b832628f08f76872ebddde9e71962d6ac9c654192c10e1eb437e1\": container with ID starting with 8336bb2d481b832628f08f76872ebddde9e71962d6ac9c654192c10e1eb437e1 not found: ID does not exist"
Oct 30 21:57:11.516873 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:11.516860    5061 scope.go:122] "RemoveContainer" containerID="a76a1f744b3dc65ae379f968708b9b3d2731dc2331f9249f74dd50d8cc36989c"
Oct 30 21:57:11.525040 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:11.525021    5061 scope.go:122] "RemoveContainer" containerID="607adf5c76fe05b67177a64fdd761144fa7b3f8d6f98cff10e60a2127b1bd0a5"
Oct 30 21:57:11.651026 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:11.650943    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-94lvp"]
Oct 30 21:57:11.698238 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:11.698201    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-94lvp"]
Oct 30 21:57:11.802347 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:11.802316    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-km4g4"]
Oct 30 21:57:11.805920 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:11.805893    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-km4g4"]
Oct 30 21:57:11.911418 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:11.911337    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-58dhn"]
Oct 30 21:57:11.914000 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:11.913976    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-58dhn"]
Oct 30 21:57:12.016788 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:12.016752    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vcstl"]
Oct 30 21:57:12.022983 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:12.022949    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vcstl"]
Oct 30 21:57:12.527914 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:12.527881    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62dfda76-164c-4a47-9328-2468648266fd" path="/var/lib/kubelet/pods/62dfda76-164c-4a47-9328-2468648266fd/volumes"
Oct 30 21:57:12.528415 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:12.528400    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="877aedb6-e77a-4cfa-9120-2650f6e39604" path="/var/lib/kubelet/pods/877aedb6-e77a-4cfa-9120-2650f6e39604/volumes"
Oct 30 21:57:12.528872 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:12.528857    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99439df0-9c24-4a03-9dff-e95af6f3f3bd" path="/var/lib/kubelet/pods/99439df0-9c24-4a03-9dff-e95af6f3f3bd/volumes"
Oct 30 21:57:12.529681 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:12.529666    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb4b4d8d-f667-4816-b351-e147ddc48224" path="/var/lib/kubelet/pods/cb4b4d8d-f667-4816-b351-e147ddc48224/volumes"
Oct 30 21:57:19.512665 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:19.512626    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a761260c-hqtc6"]
Oct 30 21:57:19.513372 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:19.513353    5061 cpu_manager.go:414] "containerMap: removing container" podUID="cb4b4d8d-f667-4816-b351-e147ddc48224" containerName="istio-proxy"
Oct 30 21:57:19.513372 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:19.513372    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb4b4d8d-f667-4816-b351-e147ddc48224" containerName="istio-proxy"
Oct 30 21:57:19.513574 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:19.513389    5061 cpu_manager.go:414] "containerMap: removing container" podUID="99439df0-9c24-4a03-9dff-e95af6f3f3bd" containerName="istio-proxy"
Oct 30 21:57:19.513574 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:19.513397    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99439df0-9c24-4a03-9dff-e95af6f3f3bd" containerName="istio-proxy"
Oct 30 21:57:19.513574 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:19.513415    5061 cpu_manager.go:414] "containerMap: removing container" podUID="62dfda76-164c-4a47-9328-2468648266fd" containerName="istio-proxy"
Oct 30 21:57:19.513574 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:19.513422    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62dfda76-164c-4a47-9328-2468648266fd" containerName="istio-proxy"
Oct 30 21:57:19.513574 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:19.513470    5061 cpu_manager.go:414] "containerMap: removing container" podUID="877aedb6-e77a-4cfa-9120-2650f6e39604" containerName="istio-proxy"
Oct 30 21:57:19.513574 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:19.513478    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="877aedb6-e77a-4cfa-9120-2650f6e39604" containerName="istio-proxy"
Oct 30 21:57:19.513815 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:19.513624    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="877aedb6-e77a-4cfa-9120-2650f6e39604" containerName="istio-proxy"
Oct 30 21:57:19.513815 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:19.513638    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb4b4d8d-f667-4816-b351-e147ddc48224" containerName="istio-proxy"
Oct 30 21:57:19.513815 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:19.513647    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="99439df0-9c24-4a03-9dff-e95af6f3f3bd" containerName="istio-proxy"
Oct 30 21:57:19.513815 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:19.513656    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="99439df0-9c24-4a03-9dff-e95af6f3f3bd" containerName="istio-proxy"
Oct 30 21:57:19.513815 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:19.513664    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="62dfda76-164c-4a47-9328-2468648266fd" containerName="istio-proxy"
Oct 30 21:57:19.518592 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:19.518572    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a761260c-hqtc6"
Oct 30 21:57:19.572330 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:19.572302    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a761260c-hqtc6"]
Oct 30 21:57:19.582923 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:19.582897    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:57:19.582923 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:19.582943    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gdmfg\""
Oct 30 21:57:19.621180 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:19.620906    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f98000ef-7846-4621-973a-ebb69d90f775-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a761260c-hqtc6\" (UID: \"f98000ef-7846-4621-973a-ebb69d90f775\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a761260c-hqtc6"
Oct 30 21:57:19.621180 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:19.620972    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f98000ef-7846-4621-973a-ebb69d90f775-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a761260c-hqtc6\" (UID: \"f98000ef-7846-4621-973a-ebb69d90f775\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a761260c-hqtc6"
Oct 30 21:57:19.621412 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:19.621221    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grcrt\" (UniqueName: \"kubernetes.io/projected/f98000ef-7846-4621-973a-ebb69d90f775-kube-api-access-grcrt\") pod \"maas-api-revoke-sub-a761260c-hqtc6\" (UID: \"f98000ef-7846-4621-973a-ebb69d90f775\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a761260c-hqtc6"
Oct 30 21:57:19.722922 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:19.722891    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f98000ef-7846-4621-973a-ebb69d90f775-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a761260c-hqtc6\" (UID: \"f98000ef-7846-4621-973a-ebb69d90f775\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a761260c-hqtc6"
Oct 30 21:57:19.723072 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:19.722937    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f98000ef-7846-4621-973a-ebb69d90f775-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a761260c-hqtc6\" (UID: \"f98000ef-7846-4621-973a-ebb69d90f775\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a761260c-hqtc6"
Oct 30 21:57:19.723072 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:19.722975    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grcrt\" (UniqueName: \"kubernetes.io/projected/f98000ef-7846-4621-973a-ebb69d90f775-kube-api-access-grcrt\") pod \"maas-api-revoke-sub-a761260c-hqtc6\" (UID: \"f98000ef-7846-4621-973a-ebb69d90f775\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a761260c-hqtc6"
Oct 30 21:57:19.723581 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:19.723521    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f98000ef-7846-4621-973a-ebb69d90f775-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a761260c-hqtc6\" (UID: \"f98000ef-7846-4621-973a-ebb69d90f775\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a761260c-hqtc6"
Oct 30 21:57:19.725226 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:19.725208    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f98000ef-7846-4621-973a-ebb69d90f775-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a761260c-hqtc6\" (UID: \"f98000ef-7846-4621-973a-ebb69d90f775\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a761260c-hqtc6"
Oct 30 21:57:19.778392 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:19.778367    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grcrt\" (UniqueName: \"kubernetes.io/projected/f98000ef-7846-4621-973a-ebb69d90f775-kube-api-access-grcrt\") pod \"maas-api-revoke-sub-a761260c-hqtc6\" (UID: \"f98000ef-7846-4621-973a-ebb69d90f775\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a761260c-hqtc6"
Oct 30 21:57:19.831412 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:19.831377    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a761260c-hqtc6"
Oct 30 21:57:20.203471 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:20.203397    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a761260c-hqtc6"]
Oct 30 21:57:20.208708 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:57:20.208682    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf98000ef_7846_4621_973a_ebb69d90f775.slice/crio-f2d5fbdc94feb24e05f1a0cecf0d50f0373c6d2866a875add59a203f135479ae WatchSource:0}: Error finding container f2d5fbdc94feb24e05f1a0cecf0d50f0373c6d2866a875add59a203f135479ae: Status 404 returned error can't find the container with id f2d5fbdc94feb24e05f1a0cecf0d50f0373c6d2866a875add59a203f135479ae
Oct 30 21:57:20.555105 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:20.555075    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a761260c-hqtc6" event={"ID":"f98000ef-7846-4621-973a-ebb69d90f775","Type":"ContainerStarted","Data":"f2d5fbdc94feb24e05f1a0cecf0d50f0373c6d2866a875add59a203f135479ae"}
Oct 30 21:57:24.666456 ip-10-0-130-60 kubenswrapper[5061]: E1030 21:57:24.666414    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 21:57:25.587568 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:25.587450    5061 generic.go:356] "Generic (PLEG): container finished" podID="f98000ef-7846-4621-973a-ebb69d90f775" containerID="2eec9d5eb1e1b23e1150255a8ef82fced17e0b7952ea0c79087871b411246476" exitCode=0
Oct 30 21:57:25.587568 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:25.587509    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a761260c-hqtc6" event={"ID":"f98000ef-7846-4621-973a-ebb69d90f775","Type":"ContainerDied","Data":"2eec9d5eb1e1b23e1150255a8ef82fced17e0b7952ea0c79087871b411246476"}
Oct 30 21:57:26.646158 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:26.645674    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-sclrt_5e74071e-0462-42b5-a05c-d90bb638478a/console-operator/2.log"
Oct 30 21:57:26.646158 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:26.645872    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-sclrt_5e74071e-0462-42b5-a05c-d90bb638478a/console-operator/2.log"
Oct 30 21:57:26.759513 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:26.759487    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a761260c-hqtc6"
Oct 30 21:57:26.903786 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:26.903705    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f98000ef-7846-4621-973a-ebb69d90f775-kube-api-access-grcrt\" (UniqueName: \"kubernetes.io/projected/f98000ef-7846-4621-973a-ebb69d90f775-kube-api-access-grcrt\") pod \"f98000ef-7846-4621-973a-ebb69d90f775\" (UID: \"f98000ef-7846-4621-973a-ebb69d90f775\") "
Oct 30 21:57:26.903786 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:26.903747    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f98000ef-7846-4621-973a-ebb69d90f775-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f98000ef-7846-4621-973a-ebb69d90f775-maas-api-openshift-ca\") pod \"f98000ef-7846-4621-973a-ebb69d90f775\" (UID: \"f98000ef-7846-4621-973a-ebb69d90f775\") "
Oct 30 21:57:26.903786 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:26.903775    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f98000ef-7846-4621-973a-ebb69d90f775-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f98000ef-7846-4621-973a-ebb69d90f775-maas-api-xks-ca\") pod \"f98000ef-7846-4621-973a-ebb69d90f775\" (UID: \"f98000ef-7846-4621-973a-ebb69d90f775\") "
Oct 30 21:57:26.904165 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:26.904129    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f98000ef-7846-4621-973a-ebb69d90f775-maas-api-openshift-ca" pod "f98000ef-7846-4621-973a-ebb69d90f775" (UID: "f98000ef-7846-4621-973a-ebb69d90f775"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:57:26.904435 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:26.904410    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f98000ef-7846-4621-973a-ebb69d90f775-maas-api-openshift-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:26.906189 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:26.906135    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f98000ef-7846-4621-973a-ebb69d90f775-maas-api-xks-ca" pod "f98000ef-7846-4621-973a-ebb69d90f775" (UID: "f98000ef-7846-4621-973a-ebb69d90f775"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:57:26.906297 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:26.906260    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f98000ef-7846-4621-973a-ebb69d90f775-kube-api-access-grcrt" pod "f98000ef-7846-4621-973a-ebb69d90f775" (UID: "f98000ef-7846-4621-973a-ebb69d90f775"). InnerVolumeSpecName "kube-api-access-grcrt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:57:27.005026 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:27.004996    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f98000ef-7846-4621-973a-ebb69d90f775-maas-api-xks-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:27.005026 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:27.005021    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grcrt\" (UniqueName: \"kubernetes.io/projected/f98000ef-7846-4621-973a-ebb69d90f775-kube-api-access-grcrt\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:27.599825 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:27.599788    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a761260c-hqtc6" event={"ID":"f98000ef-7846-4621-973a-ebb69d90f775","Type":"ContainerDied","Data":"f2d5fbdc94feb24e05f1a0cecf0d50f0373c6d2866a875add59a203f135479ae"}
Oct 30 21:57:27.599825 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:27.599808    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a761260c-hqtc6"
Oct 30 21:57:27.600028 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:27.599839    5061 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f2d5fbdc94feb24e05f1a0cecf0d50f0373c6d2866a875add59a203f135479ae"
Oct 30 21:57:50.774575 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:50.774523    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03e931f4-gbsdp"]
Oct 30 21:57:50.775115 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:50.775097    5061 cpu_manager.go:414] "containerMap: removing container" podUID="99439df0-9c24-4a03-9dff-e95af6f3f3bd" containerName="istio-proxy"
Oct 30 21:57:50.775173 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:50.775118    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99439df0-9c24-4a03-9dff-e95af6f3f3bd" containerName="istio-proxy"
Oct 30 21:57:50.775212 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:50.775178    5061 cpu_manager.go:414] "containerMap: removing container" podUID="f98000ef-7846-4621-973a-ebb69d90f775" containerName="revoke-keys"
Oct 30 21:57:50.775212 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:50.775187    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f98000ef-7846-4621-973a-ebb69d90f775" containerName="revoke-keys"
Oct 30 21:57:50.775334 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:50.775322    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="f98000ef-7846-4621-973a-ebb69d90f775" containerName="revoke-keys"
Oct 30 21:57:50.778786 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:50.778767    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03e931f4-gbsdp"
Oct 30 21:57:50.834908 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:50.834879    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03e931f4-gbsdp"]
Oct 30 21:57:50.880152 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:50.880127    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gdmfg\""
Oct 30 21:57:50.880266 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:50.880161    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:57:50.893048 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:50.893022    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkppq\" (UniqueName: \"kubernetes.io/projected/ebb4214e-0beb-474e-aa33-d9ebf9229826-kube-api-access-vkppq\") pod \"maas-api-revoke-sub-03e931f4-gbsdp\" (UID: \"ebb4214e-0beb-474e-aa33-d9ebf9229826\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03e931f4-gbsdp"
Oct 30 21:57:50.893164 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:50.893084    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebb4214e-0beb-474e-aa33-d9ebf9229826-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-03e931f4-gbsdp\" (UID: \"ebb4214e-0beb-474e-aa33-d9ebf9229826\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03e931f4-gbsdp"
Oct 30 21:57:50.893224 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:50.893204    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebb4214e-0beb-474e-aa33-d9ebf9229826-maas-api-xks-ca\") pod \"maas-api-revoke-sub-03e931f4-gbsdp\" (UID: \"ebb4214e-0beb-474e-aa33-d9ebf9229826\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03e931f4-gbsdp"
Oct 30 21:57:50.994523 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:50.994487    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkppq\" (UniqueName: \"kubernetes.io/projected/ebb4214e-0beb-474e-aa33-d9ebf9229826-kube-api-access-vkppq\") pod \"maas-api-revoke-sub-03e931f4-gbsdp\" (UID: \"ebb4214e-0beb-474e-aa33-d9ebf9229826\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03e931f4-gbsdp"
Oct 30 21:57:50.994731 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:50.994590    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebb4214e-0beb-474e-aa33-d9ebf9229826-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-03e931f4-gbsdp\" (UID: \"ebb4214e-0beb-474e-aa33-d9ebf9229826\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03e931f4-gbsdp"
Oct 30 21:57:50.994731 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:50.994655    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebb4214e-0beb-474e-aa33-d9ebf9229826-maas-api-xks-ca\") pod \"maas-api-revoke-sub-03e931f4-gbsdp\" (UID: \"ebb4214e-0beb-474e-aa33-d9ebf9229826\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03e931f4-gbsdp"
Oct 30 21:57:50.995283 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:50.995256    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebb4214e-0beb-474e-aa33-d9ebf9229826-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-03e931f4-gbsdp\" (UID: \"ebb4214e-0beb-474e-aa33-d9ebf9229826\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03e931f4-gbsdp"
Oct 30 21:57:50.996964 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:50.996945    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebb4214e-0beb-474e-aa33-d9ebf9229826-maas-api-xks-ca\") pod \"maas-api-revoke-sub-03e931f4-gbsdp\" (UID: \"ebb4214e-0beb-474e-aa33-d9ebf9229826\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03e931f4-gbsdp"
Oct 30 21:57:51.049013 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:51.048986    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkppq\" (UniqueName: \"kubernetes.io/projected/ebb4214e-0beb-474e-aa33-d9ebf9229826-kube-api-access-vkppq\") pod \"maas-api-revoke-sub-03e931f4-gbsdp\" (UID: \"ebb4214e-0beb-474e-aa33-d9ebf9229826\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03e931f4-gbsdp"
Oct 30 21:57:51.092737 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:51.092707    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03e931f4-gbsdp"
Oct 30 21:57:51.296990 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:51.296958    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-txl6d"]
Oct 30 21:57:51.301467 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:51.301408    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-txl6d"
Oct 30 21:57:51.351055 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:51.351032    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-5402dd-openshift-default-dockercfg-lstcs\""
Oct 30 21:57:51.361021 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:51.361001    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-txl6d"]
Oct 30 21:57:51.398651 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:51.398617    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbaf4a9d-78f3-449b-a2ee-3b7fe4421745-istiod-ca-cert\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-txl6d\" (UID: \"cbaf4a9d-78f3-449b-a2ee-3b7fe4421745\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-txl6d"
Oct 30 21:57:51.398836 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:51.398687    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbaf4a9d-78f3-449b-a2ee-3b7fe4421745-istio-envoy\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-txl6d\" (UID: \"cbaf4a9d-78f3-449b-a2ee-3b7fe4421745\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-txl6d"
Oct 30 21:57:51.398836 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:51.398777    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbaf4a9d-78f3-449b-a2ee-3b7fe4421745-credential-socket\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-txl6d\" (UID: \"cbaf4a9d-78f3-449b-a2ee-3b7fe4421745\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-txl6d"
Oct 30 21:57:51.398964 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:51.398899    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbaf4a9d-78f3-449b-a2ee-3b7fe4421745-workload-certs\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-txl6d\" (UID: \"cbaf4a9d-78f3-449b-a2ee-3b7fe4421745\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-txl6d"
Oct 30 21:57:51.399010 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:51.398965    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7hxv\" (UniqueName: \"kubernetes.io/projected/cbaf4a9d-78f3-449b-a2ee-3b7fe4421745-kube-api-access-b7hxv\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-txl6d\" (UID: \"cbaf4a9d-78f3-449b-a2ee-3b7fe4421745\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-txl6d"
Oct 30 21:57:51.399052 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:51.399019    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbaf4a9d-78f3-449b-a2ee-3b7fe4421745-istio-token\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-txl6d\" (UID: \"cbaf4a9d-78f3-449b-a2ee-3b7fe4421745\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-txl6d"
Oct 30 21:57:51.399176 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:51.399160    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbaf4a9d-78f3-449b-a2ee-3b7fe4421745-istio-podinfo\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-txl6d\" (UID: \"cbaf4a9d-78f3-449b-a2ee-3b7fe4421745\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-txl6d"
Oct 30 21:57:51.399217 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:51.399191    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbaf4a9d-78f3-449b-a2ee-3b7fe4421745-workload-socket\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-txl6d\" (UID: \"cbaf4a9d-78f3-449b-a2ee-3b7fe4421745\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-txl6d"
Oct 30 21:57:51.399263 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:51.399217    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbaf4a9d-78f3-449b-a2ee-3b7fe4421745-istio-data\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-txl6d\" (UID: \"cbaf4a9d-78f3-449b-a2ee-3b7fe4421745\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-txl6d"
Oct 30 21:57:51.468688 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:51.468658    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03e931f4-gbsdp"]
Oct 30 21:57:51.473651 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:57:51.473616    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebb4214e_0beb_474e_aa33_d9ebf9229826.slice/crio-45dcb0235b8cb8ad56e03d66449358028ba91a1231584d9c1e36de4294cbb190 WatchSource:0}: Error finding container 45dcb0235b8cb8ad56e03d66449358028ba91a1231584d9c1e36de4294cbb190: Status 404 returned error can't find the container with id 45dcb0235b8cb8ad56e03d66449358028ba91a1231584d9c1e36de4294cbb190
Oct 30 21:57:51.500393 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:51.500370    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbaf4a9d-78f3-449b-a2ee-3b7fe4421745-istio-podinfo\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-txl6d\" (UID: \"cbaf4a9d-78f3-449b-a2ee-3b7fe4421745\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-txl6d"
Oct 30 21:57:51.500504 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:51.500398    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbaf4a9d-78f3-449b-a2ee-3b7fe4421745-workload-socket\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-txl6d\" (UID: \"cbaf4a9d-78f3-449b-a2ee-3b7fe4421745\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-txl6d"
Oct 30 21:57:51.500504 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:51.500417    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbaf4a9d-78f3-449b-a2ee-3b7fe4421745-istio-data\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-txl6d\" (UID: \"cbaf4a9d-78f3-449b-a2ee-3b7fe4421745\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-txl6d"
Oct 30 21:57:51.500504 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:51.500461    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbaf4a9d-78f3-449b-a2ee-3b7fe4421745-istiod-ca-cert\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-txl6d\" (UID: \"cbaf4a9d-78f3-449b-a2ee-3b7fe4421745\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-txl6d"
Oct 30 21:57:51.500504 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:51.500496    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbaf4a9d-78f3-449b-a2ee-3b7fe4421745-istio-envoy\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-txl6d\" (UID: \"cbaf4a9d-78f3-449b-a2ee-3b7fe4421745\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-txl6d"
Oct 30 21:57:51.500749 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:51.500546    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbaf4a9d-78f3-449b-a2ee-3b7fe4421745-credential-socket\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-txl6d\" (UID: \"cbaf4a9d-78f3-449b-a2ee-3b7fe4421745\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-txl6d"
Oct 30 21:57:51.500749 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:51.500644    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbaf4a9d-78f3-449b-a2ee-3b7fe4421745-workload-certs\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-txl6d\" (UID: \"cbaf4a9d-78f3-449b-a2ee-3b7fe4421745\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-txl6d"
Oct 30 21:57:51.500749 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:51.500677    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7hxv\" (UniqueName: \"kubernetes.io/projected/cbaf4a9d-78f3-449b-a2ee-3b7fe4421745-kube-api-access-b7hxv\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-txl6d\" (UID: \"cbaf4a9d-78f3-449b-a2ee-3b7fe4421745\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-txl6d"
Oct 30 21:57:51.500749 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:51.500731    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbaf4a9d-78f3-449b-a2ee-3b7fe4421745-istio-token\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-txl6d\" (UID: \"cbaf4a9d-78f3-449b-a2ee-3b7fe4421745\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-txl6d"
Oct 30 21:57:51.500990 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:51.500966    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbaf4a9d-78f3-449b-a2ee-3b7fe4421745-workload-socket\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-txl6d\" (UID: \"cbaf4a9d-78f3-449b-a2ee-3b7fe4421745\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-txl6d"
Oct 30 21:57:51.501053 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:51.501033    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbaf4a9d-78f3-449b-a2ee-3b7fe4421745-credential-socket\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-txl6d\" (UID: \"cbaf4a9d-78f3-449b-a2ee-3b7fe4421745\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-txl6d"
Oct 30 21:57:51.501244 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:51.501224    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbaf4a9d-78f3-449b-a2ee-3b7fe4421745-istio-data\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-txl6d\" (UID: \"cbaf4a9d-78f3-449b-a2ee-3b7fe4421745\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-txl6d"
Oct 30 21:57:51.501316 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:51.501248    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbaf4a9d-78f3-449b-a2ee-3b7fe4421745-workload-certs\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-txl6d\" (UID: \"cbaf4a9d-78f3-449b-a2ee-3b7fe4421745\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-txl6d"
Oct 30 21:57:51.501316 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:51.501249    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbaf4a9d-78f3-449b-a2ee-3b7fe4421745-istiod-ca-cert\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-txl6d\" (UID: \"cbaf4a9d-78f3-449b-a2ee-3b7fe4421745\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-txl6d"
Oct 30 21:57:51.502733 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:51.502712    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbaf4a9d-78f3-449b-a2ee-3b7fe4421745-istio-envoy\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-txl6d\" (UID: \"cbaf4a9d-78f3-449b-a2ee-3b7fe4421745\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-txl6d"
Oct 30 21:57:51.503007 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:51.502989    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbaf4a9d-78f3-449b-a2ee-3b7fe4421745-istio-podinfo\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-txl6d\" (UID: \"cbaf4a9d-78f3-449b-a2ee-3b7fe4421745\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-txl6d"
Oct 30 21:57:51.555305 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:51.555249    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7hxv\" (UniqueName: \"kubernetes.io/projected/cbaf4a9d-78f3-449b-a2ee-3b7fe4421745-kube-api-access-b7hxv\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-txl6d\" (UID: \"cbaf4a9d-78f3-449b-a2ee-3b7fe4421745\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-txl6d"
Oct 30 21:57:51.555411 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:51.555394    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbaf4a9d-78f3-449b-a2ee-3b7fe4421745-istio-token\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-txl6d\" (UID: \"cbaf4a9d-78f3-449b-a2ee-3b7fe4421745\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-txl6d"
Oct 30 21:57:51.615553 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:51.615518    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-txl6d"
Oct 30 21:57:51.753378 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:51.753345    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03e931f4-gbsdp" event={"ID":"ebb4214e-0beb-474e-aa33-d9ebf9229826","Type":"ContainerStarted","Data":"f40061f3bb6d6d4ae40d5f5c8b7be5223439bfb08a88c1711c2963bce852e263"}
Oct 30 21:57:51.753508 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:51.753408    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03e931f4-gbsdp" event={"ID":"ebb4214e-0beb-474e-aa33-d9ebf9229826","Type":"ContainerStarted","Data":"45dcb0235b8cb8ad56e03d66449358028ba91a1231584d9c1e36de4294cbb190"}
Oct 30 21:57:51.862623 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:51.862492    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03e931f4-gbsdp" podStartSLOduration=1.862479949 podStartE2EDuration="1.862479949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:57:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:57:51.862137428 +0000 UTC m=+925.474907185" watchObservedRunningTime="2026-10-30 21:57:51.862479949 +0000 UTC m=+925.475249701"
Oct 30 21:57:52.046803 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:52.046765    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-txl6d"]
Oct 30 21:57:52.054245 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:57:52.054203    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbaf4a9d_78f3_449b_a2ee_3b7fe4421745.slice/crio-d807d14702d3650a9e4d907370d1c0fd829bf89174b1176ffd51cc9d74612ad4 WatchSource:0}: Error finding container d807d14702d3650a9e4d907370d1c0fd829bf89174b1176ffd51cc9d74612ad4: Status 404 returned error can't find the container with id d807d14702d3650a9e4d907370d1c0fd829bf89174b1176ffd51cc9d74612ad4
Oct 30 21:57:52.056760 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:52.056725    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:57:52.056827 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:52.056804    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:57:52.056867 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:52.056843    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:57:52.760754 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:52.760709    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-txl6d" event={"ID":"cbaf4a9d-78f3-449b-a2ee-3b7fe4421745","Type":"ContainerStarted","Data":"7b168911b0017899b9cdfe2f5a400ad248031af89a9397a7f55f1aa93319860f"}
Oct 30 21:57:52.760954 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:52.760787    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-txl6d" event={"ID":"cbaf4a9d-78f3-449b-a2ee-3b7fe4421745","Type":"ContainerStarted","Data":"d807d14702d3650a9e4d907370d1c0fd829bf89174b1176ffd51cc9d74612ad4"}
Oct 30 21:57:52.762255 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:52.762216    5061 generic.go:356] "Generic (PLEG): container finished" podID="ebb4214e-0beb-474e-aa33-d9ebf9229826" containerID="f40061f3bb6d6d4ae40d5f5c8b7be5223439bfb08a88c1711c2963bce852e263" exitCode=0
Oct 30 21:57:52.762376 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:52.762303    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03e931f4-gbsdp" event={"ID":"ebb4214e-0beb-474e-aa33-d9ebf9229826","Type":"ContainerDied","Data":"f40061f3bb6d6d4ae40d5f5c8b7be5223439bfb08a88c1711c2963bce852e263"}
Oct 30 21:57:52.894979 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:52.894929    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-txl6d" podStartSLOduration=1.894916066 podStartE2EDuration="1.894916066s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:57:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:57:52.894018493 +0000 UTC m=+926.506788251" watchObservedRunningTime="2026-10-30 21:57:52.894916066 +0000 UTC m=+926.507685824"
Oct 30 21:57:53.095418 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:53.095384    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5zvwr"]
Oct 30 21:57:53.099858 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:53.099839    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5zvwr"
Oct 30 21:57:53.137472 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:53.137431    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf-istio-podinfo\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5zvwr\" (UID: \"3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5zvwr"
Oct 30 21:57:53.137472 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:53.137469    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf-workload-socket\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5zvwr\" (UID: \"3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5zvwr"
Oct 30 21:57:53.137777 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:53.137591    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf-workload-certs\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5zvwr\" (UID: \"3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5zvwr"
Oct 30 21:57:53.137777 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:53.137636    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf-istio-token\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5zvwr\" (UID: \"3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5zvwr"
Oct 30 21:57:53.137777 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:53.137664    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ht4x\" (UniqueName: \"kubernetes.io/projected/3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf-kube-api-access-7ht4x\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5zvwr\" (UID: \"3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5zvwr"
Oct 30 21:57:53.137932 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:53.137784    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf-istiod-ca-cert\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5zvwr\" (UID: \"3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5zvwr"
Oct 30 21:57:53.137932 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:53.137811    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf-istio-envoy\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5zvwr\" (UID: \"3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5zvwr"
Oct 30 21:57:53.137932 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:53.137836    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf-credential-socket\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5zvwr\" (UID: \"3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5zvwr"
Oct 30 21:57:53.137932 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:53.137866    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf-istio-data\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5zvwr\" (UID: \"3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5zvwr"
Oct 30 21:57:53.162645 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:53.162620    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5zvwr"]
Oct 30 21:57:53.238561 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:53.238487    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf-istio-podinfo\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5zvwr\" (UID: \"3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5zvwr"
Oct 30 21:57:53.238749 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:53.238568    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf-workload-socket\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5zvwr\" (UID: \"3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5zvwr"
Oct 30 21:57:53.238749 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:53.238635    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf-workload-certs\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5zvwr\" (UID: \"3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5zvwr"
Oct 30 21:57:53.238749 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:53.238682    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf-istio-token\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5zvwr\" (UID: \"3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5zvwr"
Oct 30 21:57:53.238749 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:53.238724    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ht4x\" (UniqueName: \"kubernetes.io/projected/3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf-kube-api-access-7ht4x\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5zvwr\" (UID: \"3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5zvwr"
Oct 30 21:57:53.238985 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:53.238822    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf-istiod-ca-cert\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5zvwr\" (UID: \"3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5zvwr"
Oct 30 21:57:53.238985 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:53.238852    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf-istio-envoy\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5zvwr\" (UID: \"3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5zvwr"
Oct 30 21:57:53.239094 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:53.239002    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf-workload-socket\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5zvwr\" (UID: \"3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5zvwr"
Oct 30 21:57:53.239094 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:53.239011    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf-credential-socket\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5zvwr\" (UID: \"3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5zvwr"
Oct 30 21:57:53.239094 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:53.239054    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf-workload-certs\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5zvwr\" (UID: \"3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5zvwr"
Oct 30 21:57:53.239094 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:53.239087    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf-istio-data\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5zvwr\" (UID: \"3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5zvwr"
Oct 30 21:57:53.239332 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:53.239310    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf-credential-socket\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5zvwr\" (UID: \"3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5zvwr"
Oct 30 21:57:53.239459 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:53.239419    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf-istio-data\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5zvwr\" (UID: \"3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5zvwr"
Oct 30 21:57:53.239585 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:53.239564    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf-istiod-ca-cert\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5zvwr\" (UID: \"3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5zvwr"
Oct 30 21:57:53.241069 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:53.241041    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf-istio-envoy\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5zvwr\" (UID: \"3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5zvwr"
Oct 30 21:57:53.241302 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:53.241282    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf-istio-podinfo\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5zvwr\" (UID: \"3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5zvwr"
Oct 30 21:57:53.295021 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:53.294991    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf-istio-token\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5zvwr\" (UID: \"3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5zvwr"
Oct 30 21:57:53.295562 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:53.295540    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ht4x\" (UniqueName: \"kubernetes.io/projected/3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf-kube-api-access-7ht4x\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5zvwr\" (UID: \"3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5zvwr"
Oct 30 21:57:53.337547 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:53.337496    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-rh8lc"]
Oct 30 21:57:53.342719 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:53.342698    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-rh8lc"
Oct 30 21:57:53.390918 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:53.390892    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-rh8lc"]
Oct 30 21:57:53.393244 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:53.393216    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-a07620-openshift-default-dockercfg-p8t9r\""
Oct 30 21:57:53.426562 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:53.425732    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5zvwr"
Oct 30 21:57:53.441385 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:53.441355    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d238b910-dd97-4901-8e16-274b063b66ec-istio-token\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-rh8lc\" (UID: \"d238b910-dd97-4901-8e16-274b063b66ec\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-rh8lc"
Oct 30 21:57:53.441521 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:53.441505    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d238b910-dd97-4901-8e16-274b063b66ec-workload-socket\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-rh8lc\" (UID: \"d238b910-dd97-4901-8e16-274b063b66ec\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-rh8lc"
Oct 30 21:57:53.441629 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:53.441606    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d238b910-dd97-4901-8e16-274b063b66ec-istio-data\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-rh8lc\" (UID: \"d238b910-dd97-4901-8e16-274b063b66ec\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-rh8lc"
Oct 30 21:57:53.441690 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:53.441672    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krfrm\" (UniqueName: \"kubernetes.io/projected/d238b910-dd97-4901-8e16-274b063b66ec-kube-api-access-krfrm\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-rh8lc\" (UID: \"d238b910-dd97-4901-8e16-274b063b66ec\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-rh8lc"
Oct 30 21:57:53.441745 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:53.441702    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d238b910-dd97-4901-8e16-274b063b66ec-workload-certs\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-rh8lc\" (UID: \"d238b910-dd97-4901-8e16-274b063b66ec\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-rh8lc"
Oct 30 21:57:53.441798 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:53.441758    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d238b910-dd97-4901-8e16-274b063b66ec-credential-socket\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-rh8lc\" (UID: \"d238b910-dd97-4901-8e16-274b063b66ec\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-rh8lc"
Oct 30 21:57:53.441901 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:53.441832    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d238b910-dd97-4901-8e16-274b063b66ec-istio-podinfo\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-rh8lc\" (UID: \"d238b910-dd97-4901-8e16-274b063b66ec\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-rh8lc"
Oct 30 21:57:53.441901 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:53.441862    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d238b910-dd97-4901-8e16-274b063b66ec-istiod-ca-cert\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-rh8lc\" (UID: \"d238b910-dd97-4901-8e16-274b063b66ec\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-rh8lc"
Oct 30 21:57:53.442003 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:53.441907    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d238b910-dd97-4901-8e16-274b063b66ec-istio-envoy\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-rh8lc\" (UID: \"d238b910-dd97-4901-8e16-274b063b66ec\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-rh8lc"
Oct 30 21:57:53.544396 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:53.542960    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krfrm\" (UniqueName: \"kubernetes.io/projected/d238b910-dd97-4901-8e16-274b063b66ec-kube-api-access-krfrm\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-rh8lc\" (UID: \"d238b910-dd97-4901-8e16-274b063b66ec\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-rh8lc"
Oct 30 21:57:53.544396 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:53.542993    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d238b910-dd97-4901-8e16-274b063b66ec-workload-certs\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-rh8lc\" (UID: \"d238b910-dd97-4901-8e16-274b063b66ec\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-rh8lc"
Oct 30 21:57:53.544396 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:53.543012    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d238b910-dd97-4901-8e16-274b063b66ec-credential-socket\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-rh8lc\" (UID: \"d238b910-dd97-4901-8e16-274b063b66ec\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-rh8lc"
Oct 30 21:57:53.544396 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:53.543101    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d238b910-dd97-4901-8e16-274b063b66ec-istio-podinfo\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-rh8lc\" (UID: \"d238b910-dd97-4901-8e16-274b063b66ec\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-rh8lc"
Oct 30 21:57:53.544396 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:53.543136    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d238b910-dd97-4901-8e16-274b063b66ec-istiod-ca-cert\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-rh8lc\" (UID: \"d238b910-dd97-4901-8e16-274b063b66ec\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-rh8lc"
Oct 30 21:57:53.544396 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:53.543162    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d238b910-dd97-4901-8e16-274b063b66ec-istio-envoy\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-rh8lc\" (UID: \"d238b910-dd97-4901-8e16-274b063b66ec\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-rh8lc"
Oct 30 21:57:53.544396 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:53.543219    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d238b910-dd97-4901-8e16-274b063b66ec-istio-token\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-rh8lc\" (UID: \"d238b910-dd97-4901-8e16-274b063b66ec\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-rh8lc"
Oct 30 21:57:53.544396 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:53.543342    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d238b910-dd97-4901-8e16-274b063b66ec-workload-socket\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-rh8lc\" (UID: \"d238b910-dd97-4901-8e16-274b063b66ec\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-rh8lc"
Oct 30 21:57:53.544396 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:53.543402    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d238b910-dd97-4901-8e16-274b063b66ec-istio-data\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-rh8lc\" (UID: \"d238b910-dd97-4901-8e16-274b063b66ec\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-rh8lc"
Oct 30 21:57:53.544396 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:53.543550    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d238b910-dd97-4901-8e16-274b063b66ec-workload-certs\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-rh8lc\" (UID: \"d238b910-dd97-4901-8e16-274b063b66ec\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-rh8lc"
Oct 30 21:57:53.544396 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:53.543761    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d238b910-dd97-4901-8e16-274b063b66ec-istio-data\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-rh8lc\" (UID: \"d238b910-dd97-4901-8e16-274b063b66ec\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-rh8lc"
Oct 30 21:57:53.544396 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:53.543785    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d238b910-dd97-4901-8e16-274b063b66ec-workload-socket\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-rh8lc\" (UID: \"d238b910-dd97-4901-8e16-274b063b66ec\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-rh8lc"
Oct 30 21:57:53.544396 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:53.544303    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d238b910-dd97-4901-8e16-274b063b66ec-credential-socket\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-rh8lc\" (UID: \"d238b910-dd97-4901-8e16-274b063b66ec\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-rh8lc"
Oct 30 21:57:53.545608 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:53.545561    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d238b910-dd97-4901-8e16-274b063b66ec-istiod-ca-cert\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-rh8lc\" (UID: \"d238b910-dd97-4901-8e16-274b063b66ec\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-rh8lc"
Oct 30 21:57:53.546858 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:53.546510    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d238b910-dd97-4901-8e16-274b063b66ec-istio-envoy\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-rh8lc\" (UID: \"d238b910-dd97-4901-8e16-274b063b66ec\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-rh8lc"
Oct 30 21:57:53.546858 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:53.546761    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d238b910-dd97-4901-8e16-274b063b66ec-istio-podinfo\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-rh8lc\" (UID: \"d238b910-dd97-4901-8e16-274b063b66ec\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-rh8lc"
Oct 30 21:57:53.597674 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:53.597647    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d238b910-dd97-4901-8e16-274b063b66ec-istio-token\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-rh8lc\" (UID: \"d238b910-dd97-4901-8e16-274b063b66ec\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-rh8lc"
Oct 30 21:57:53.597826 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:53.597807    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krfrm\" (UniqueName: \"kubernetes.io/projected/d238b910-dd97-4901-8e16-274b063b66ec-kube-api-access-krfrm\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-rh8lc\" (UID: \"d238b910-dd97-4901-8e16-274b063b66ec\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-rh8lc"
Oct 30 21:57:53.632718 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:53.632652    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-txl6d"
Oct 30 21:57:53.634343 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:53.634321    5061 patch_prober.go:29] interesting pod/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-txl6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Oct 30 21:57:53.634405 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:53.634370    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-txl6d" podUID="cbaf4a9d-78f3-449b-a2ee-3b7fe4421745" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Oct 30 21:57:53.657221 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:53.657200    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-rh8lc"
Oct 30 21:57:53.859666 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:53.859567    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5zvwr"]
Oct 30 21:57:53.866825 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:53.866786    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:57:53.866940 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:53.866876    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:57:53.866940 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:53.866919    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:57:54.003780 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:54.003752    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03e931f4-gbsdp"
Oct 30 21:57:54.050837 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:54.049951    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ebb4214e-0beb-474e-aa33-d9ebf9229826-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebb4214e-0beb-474e-aa33-d9ebf9229826-maas-api-xks-ca\") pod \"ebb4214e-0beb-474e-aa33-d9ebf9229826\" (UID: \"ebb4214e-0beb-474e-aa33-d9ebf9229826\") "
Oct 30 21:57:54.050837 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:54.050023    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebb4214e-0beb-474e-aa33-d9ebf9229826-kube-api-access-vkppq\" (UniqueName: \"kubernetes.io/projected/ebb4214e-0beb-474e-aa33-d9ebf9229826-kube-api-access-vkppq\") pod \"ebb4214e-0beb-474e-aa33-d9ebf9229826\" (UID: \"ebb4214e-0beb-474e-aa33-d9ebf9229826\") "
Oct 30 21:57:54.050837 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:54.050097    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ebb4214e-0beb-474e-aa33-d9ebf9229826-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebb4214e-0beb-474e-aa33-d9ebf9229826-maas-api-openshift-ca\") pod \"ebb4214e-0beb-474e-aa33-d9ebf9229826\" (UID: \"ebb4214e-0beb-474e-aa33-d9ebf9229826\") "
Oct 30 21:57:54.051282 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:54.051238    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebb4214e-0beb-474e-aa33-d9ebf9229826-maas-api-openshift-ca" pod "ebb4214e-0beb-474e-aa33-d9ebf9229826" (UID: "ebb4214e-0beb-474e-aa33-d9ebf9229826"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:57:54.056599 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:54.056572    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebb4214e-0beb-474e-aa33-d9ebf9229826-maas-api-xks-ca" pod "ebb4214e-0beb-474e-aa33-d9ebf9229826" (UID: "ebb4214e-0beb-474e-aa33-d9ebf9229826"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:57:54.057793 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:54.057766    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebb4214e-0beb-474e-aa33-d9ebf9229826-kube-api-access-vkppq" pod "ebb4214e-0beb-474e-aa33-d9ebf9229826" (UID: "ebb4214e-0beb-474e-aa33-d9ebf9229826"). InnerVolumeSpecName "kube-api-access-vkppq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:57:54.059188 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:54.059106    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-rh8lc"]
Oct 30 21:57:54.066845 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:54.064205    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:57:54.066845 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:54.064266    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:57:54.066845 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:54.064298    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:57:54.151926 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:54.151843    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebb4214e-0beb-474e-aa33-d9ebf9229826-maas-api-xks-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:54.151926 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:54.151871    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkppq\" (UniqueName: \"kubernetes.io/projected/ebb4214e-0beb-474e-aa33-d9ebf9229826-kube-api-access-vkppq\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:54.151926 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:54.151890    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebb4214e-0beb-474e-aa33-d9ebf9229826-maas-api-openshift-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:54.616082 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:54.616050    5061 patch_prober.go:29] interesting pod/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-txl6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Oct 30 21:57:54.616352 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:54.616314    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-txl6d" podUID="cbaf4a9d-78f3-449b-a2ee-3b7fe4421745" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Oct 30 21:57:54.690666 ip-10-0-130-60 kubenswrapper[5061]: E1030 21:57:54.690631    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 21:57:54.791629 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:54.791597    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03e931f4-gbsdp"
Oct 30 21:57:54.791794 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:54.791637    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03e931f4-gbsdp" event={"ID":"ebb4214e-0beb-474e-aa33-d9ebf9229826","Type":"ContainerDied","Data":"45dcb0235b8cb8ad56e03d66449358028ba91a1231584d9c1e36de4294cbb190"}
Oct 30 21:57:54.791794 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:54.791689    5061 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="45dcb0235b8cb8ad56e03d66449358028ba91a1231584d9c1e36de4294cbb190"
Oct 30 21:57:54.802198 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:54.802154    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5zvwr" event={"ID":"3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf","Type":"ContainerStarted","Data":"bb5f3d13f979cbf2216cd47e9e0c6af992a4f3cda415ac5fd12663e716e9d48c"}
Oct 30 21:57:54.802350 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:54.802236    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5zvwr" event={"ID":"3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf","Type":"ContainerStarted","Data":"df7501a516cfcb486cb7e48b31a0b42c1a35d9a7520fd78f0ec01a2cf652f43d"}
Oct 30 21:57:54.804138 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:54.804110    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-rh8lc" event={"ID":"d238b910-dd97-4901-8e16-274b063b66ec","Type":"ContainerStarted","Data":"a0d98cfa8080dbbca0afe6db33b277854ae57aea174343d33e4a310c664dc2aa"}
Oct 30 21:57:54.804267 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:54.804156    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-rh8lc" event={"ID":"d238b910-dd97-4901-8e16-274b063b66ec","Type":"ContainerStarted","Data":"887e4f269583fbf7e5e8c18034308fa557ac836cab26b11b7bee6c8f7470502c"}
Oct 30 21:57:55.151550 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:55.150625    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5zvwr" podStartSLOduration=2.150603239 podStartE2EDuration="2.150603239s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:57:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:57:54.995293677 +0000 UTC m=+928.608063435" watchObservedRunningTime="2026-10-30 21:57:55.150603239 +0000 UTC m=+928.763372997"
Oct 30 21:57:55.430668 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:55.430580    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5zvwr"
Oct 30 21:57:55.445559 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:55.443681    5061 patch_prober.go:29] interesting pod/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5zvwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Oct 30 21:57:55.445559 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:55.443764    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5zvwr" podUID="3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Oct 30 21:57:55.623553 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:55.620009    5061 patch_prober.go:29] interesting pod/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-txl6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Oct 30 21:57:55.623553 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:55.620082    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-txl6d" podUID="cbaf4a9d-78f3-449b-a2ee-3b7fe4421745" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Oct 30 21:57:55.658551 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:55.657928    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-rh8lc"
Oct 30 21:57:55.669562 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:55.668655    5061 patch_prober.go:29] interesting pod/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-rh8lc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Oct 30 21:57:55.669562 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:55.668720    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-rh8lc" podUID="d238b910-dd97-4901-8e16-274b063b66ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Oct 30 21:57:56.481703 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:56.481662    5061 patch_prober.go:29] interesting pod/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5zvwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Oct 30 21:57:56.482301 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:56.482263    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5zvwr" podUID="3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Oct 30 21:57:56.628555 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:56.620204    5061 patch_prober.go:29] interesting pod/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-txl6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Oct 30 21:57:56.628555 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:56.620264    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-txl6d" podUID="cbaf4a9d-78f3-449b-a2ee-3b7fe4421745" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Oct 30 21:57:56.658263 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:56.658223    5061 patch_prober.go:29] interesting pod/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-rh8lc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Oct 30 21:57:56.658574 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:56.658519    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-rh8lc" podUID="d238b910-dd97-4901-8e16-274b063b66ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Oct 30 21:57:57.426916 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:57.426876    5061 patch_prober.go:29] interesting pod/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5zvwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Oct 30 21:57:57.427139 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:57.426936    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5zvwr" podUID="3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Oct 30 21:57:57.619922 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:57.619882    5061 patch_prober.go:29] interesting pod/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-txl6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Oct 30 21:57:57.620564 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:57.620506    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-txl6d" podUID="cbaf4a9d-78f3-449b-a2ee-3b7fe4421745" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Oct 30 21:57:57.657933 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:57.657894    5061 patch_prober.go:29] interesting pod/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-rh8lc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Oct 30 21:57:57.658203 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:57.658174    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-rh8lc" podUID="d238b910-dd97-4901-8e16-274b063b66ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Oct 30 21:57:58.426881 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:58.426844    5061 patch_prober.go:29] interesting pod/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5zvwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Oct 30 21:57:58.427165 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:58.427136    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5zvwr" podUID="3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Oct 30 21:57:58.616891 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:58.616855    5061 patch_prober.go:29] interesting pod/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-txl6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Oct 30 21:57:58.617143 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:58.617114    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-txl6d" podUID="cbaf4a9d-78f3-449b-a2ee-3b7fe4421745" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Oct 30 21:57:58.657975 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:58.657938    5061 patch_prober.go:29] interesting pod/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-rh8lc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Oct 30 21:57:58.658615 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:58.658583    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-rh8lc" podUID="d238b910-dd97-4901-8e16-274b063b66ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Oct 30 21:57:59.429925 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:59.429886    5061 patch_prober.go:29] interesting pod/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5zvwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Oct 30 21:57:59.430219 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:59.430192    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5zvwr" podUID="3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Oct 30 21:57:59.633860 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:59.633826    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-txl6d"
Oct 30 21:57:59.635391 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:59.635370    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-txl6d"
Oct 30 21:57:59.638744 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:59.638723    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-txl6d"
Oct 30 21:57:59.657913 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:59.657883    5061 patch_prober.go:29] interesting pod/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-rh8lc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Oct 30 21:57:59.658120 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:59.658094    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-rh8lc" podUID="d238b910-dd97-4901-8e16-274b063b66ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Oct 30 21:57:59.845922 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:57:59.845853    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-rh8lc" podStartSLOduration=6.845821881 podStartE2EDuration="6.845821881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:57:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:57:55.134848856 +0000 UTC m=+928.747618643" watchObservedRunningTime="2026-10-30 21:57:59.845821881 +0000 UTC m=+933.458591640"
Oct 30 21:58:00.427192 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:00.427152    5061 patch_prober.go:29] interesting pod/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5zvwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Oct 30 21:58:00.427378 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:00.427220    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5zvwr" podUID="3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Oct 30 21:58:00.657896 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:00.657863    5061 patch_prober.go:29] interesting pod/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-rh8lc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Oct 30 21:58:00.658093 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:00.657946    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-rh8lc" podUID="d238b910-dd97-4901-8e16-274b063b66ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Oct 30 21:58:01.430630 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:01.430601    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5zvwr"
Oct 30 21:58:01.431109 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:01.430852    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5zvwr"
Oct 30 21:58:01.431581 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:01.431562    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5zvwr"
Oct 30 21:58:01.662106 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:01.662080    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-rh8lc"
Oct 30 21:58:01.662368 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:01.662348    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-rh8lc"
Oct 30 21:58:01.663144 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:01.663125    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-rh8lc"
Oct 30 21:58:04.753852 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:04.753813    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-343fe207-chkvn"]
Oct 30 21:58:04.754730 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:04.754662    5061 cpu_manager.go:414] "containerMap: removing container" podUID="ebb4214e-0beb-474e-aa33-d9ebf9229826" containerName="revoke-keys"
Oct 30 21:58:04.754730 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:04.754686    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebb4214e-0beb-474e-aa33-d9ebf9229826" containerName="revoke-keys"
Oct 30 21:58:04.754990 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:04.754837    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebb4214e-0beb-474e-aa33-d9ebf9229826" containerName="revoke-keys"
Oct 30 21:58:04.770843 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:04.770801    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-343fe207-chkvn"
Oct 30 21:58:04.812237 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:04.812207    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-343fe207-chkvn"]
Oct 30 21:58:04.823548 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:04.823484    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:58:04.845122 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:04.845093    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gdmfg\""
Oct 30 21:58:04.912446 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:04.912413    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/113abc93-9162-4924-a232-29bceb7ef71d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-343fe207-chkvn\" (UID: \"113abc93-9162-4924-a232-29bceb7ef71d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-343fe207-chkvn"
Oct 30 21:58:04.912723 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:04.912507    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/113abc93-9162-4924-a232-29bceb7ef71d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-343fe207-chkvn\" (UID: \"113abc93-9162-4924-a232-29bceb7ef71d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-343fe207-chkvn"
Oct 30 21:58:04.912723 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:04.912570    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5q942\" (UniqueName: \"kubernetes.io/projected/113abc93-9162-4924-a232-29bceb7ef71d-kube-api-access-5q942\") pod \"maas-api-revoke-sub-343fe207-chkvn\" (UID: \"113abc93-9162-4924-a232-29bceb7ef71d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-343fe207-chkvn"
Oct 30 21:58:05.013645 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:05.013563    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/113abc93-9162-4924-a232-29bceb7ef71d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-343fe207-chkvn\" (UID: \"113abc93-9162-4924-a232-29bceb7ef71d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-343fe207-chkvn"
Oct 30 21:58:05.013820 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:05.013633    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/113abc93-9162-4924-a232-29bceb7ef71d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-343fe207-chkvn\" (UID: \"113abc93-9162-4924-a232-29bceb7ef71d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-343fe207-chkvn"
Oct 30 21:58:05.013820 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:05.013699    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5q942\" (UniqueName: \"kubernetes.io/projected/113abc93-9162-4924-a232-29bceb7ef71d-kube-api-access-5q942\") pod \"maas-api-revoke-sub-343fe207-chkvn\" (UID: \"113abc93-9162-4924-a232-29bceb7ef71d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-343fe207-chkvn"
Oct 30 21:58:05.014486 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:05.014452    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/113abc93-9162-4924-a232-29bceb7ef71d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-343fe207-chkvn\" (UID: \"113abc93-9162-4924-a232-29bceb7ef71d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-343fe207-chkvn"
Oct 30 21:58:05.016269 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:05.016244    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/113abc93-9162-4924-a232-29bceb7ef71d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-343fe207-chkvn\" (UID: \"113abc93-9162-4924-a232-29bceb7ef71d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-343fe207-chkvn"
Oct 30 21:58:05.091549 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:05.090163    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5q942\" (UniqueName: \"kubernetes.io/projected/113abc93-9162-4924-a232-29bceb7ef71d-kube-api-access-5q942\") pod \"maas-api-revoke-sub-343fe207-chkvn\" (UID: \"113abc93-9162-4924-a232-29bceb7ef71d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-343fe207-chkvn"
Oct 30 21:58:05.387273 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:05.387236    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-343fe207-chkvn"
Oct 30 21:58:05.814488 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:05.814384    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-343fe207-chkvn"]
Oct 30 21:58:05.826571 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:58:05.824232    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod113abc93_9162_4924_a232_29bceb7ef71d.slice/crio-45fe2674d21c942721e2d9b4a3eacf097f54bb7669d4846f7f3c4eaa74cba488 WatchSource:0}: Error finding container 45fe2674d21c942721e2d9b4a3eacf097f54bb7669d4846f7f3c4eaa74cba488: Status 404 returned error can't find the container with id 45fe2674d21c942721e2d9b4a3eacf097f54bb7669d4846f7f3c4eaa74cba488
Oct 30 21:58:05.944503 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:05.944366    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-343fe207-chkvn" event={"ID":"113abc93-9162-4924-a232-29bceb7ef71d","Type":"ContainerStarted","Data":"45fe2674d21c942721e2d9b4a3eacf097f54bb7669d4846f7f3c4eaa74cba488"}
Oct 30 21:58:06.951092 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:06.951063    5061 generic.go:356] "Generic (PLEG): container finished" podID="113abc93-9162-4924-a232-29bceb7ef71d" containerID="811260f68d964cd95694cf250e29fcf0451ff7ee7de171e40a91454a01111ece" exitCode=0
Oct 30 21:58:06.951438 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:06.951145    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-343fe207-chkvn" event={"ID":"113abc93-9162-4924-a232-29bceb7ef71d","Type":"ContainerDied","Data":"811260f68d964cd95694cf250e29fcf0451ff7ee7de171e40a91454a01111ece"}
Oct 30 21:58:08.124052 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:08.124028    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-343fe207-chkvn"
Oct 30 21:58:08.266837 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:08.266722    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/113abc93-9162-4924-a232-29bceb7ef71d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/113abc93-9162-4924-a232-29bceb7ef71d-maas-api-openshift-ca\") pod \"113abc93-9162-4924-a232-29bceb7ef71d\" (UID: \"113abc93-9162-4924-a232-29bceb7ef71d\") "
Oct 30 21:58:08.266999 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:08.266854    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/113abc93-9162-4924-a232-29bceb7ef71d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/113abc93-9162-4924-a232-29bceb7ef71d-maas-api-xks-ca\") pod \"113abc93-9162-4924-a232-29bceb7ef71d\" (UID: \"113abc93-9162-4924-a232-29bceb7ef71d\") "
Oct 30 21:58:08.266999 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:08.266912    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/113abc93-9162-4924-a232-29bceb7ef71d-kube-api-access-5q942\" (UniqueName: \"kubernetes.io/projected/113abc93-9162-4924-a232-29bceb7ef71d-kube-api-access-5q942\") pod \"113abc93-9162-4924-a232-29bceb7ef71d\" (UID: \"113abc93-9162-4924-a232-29bceb7ef71d\") "
Oct 30 21:58:08.267182 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:08.267155    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/113abc93-9162-4924-a232-29bceb7ef71d-maas-api-openshift-ca" pod "113abc93-9162-4924-a232-29bceb7ef71d" (UID: "113abc93-9162-4924-a232-29bceb7ef71d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:58:08.267495 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:08.267461    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/113abc93-9162-4924-a232-29bceb7ef71d-maas-api-openshift-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:08.269041 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:08.269018    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/113abc93-9162-4924-a232-29bceb7ef71d-kube-api-access-5q942" pod "113abc93-9162-4924-a232-29bceb7ef71d" (UID: "113abc93-9162-4924-a232-29bceb7ef71d"). InnerVolumeSpecName "kube-api-access-5q942". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:58:08.269635 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:08.269616    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/113abc93-9162-4924-a232-29bceb7ef71d-maas-api-xks-ca" pod "113abc93-9162-4924-a232-29bceb7ef71d" (UID: "113abc93-9162-4924-a232-29bceb7ef71d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:58:08.368177 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:08.368147    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/113abc93-9162-4924-a232-29bceb7ef71d-maas-api-xks-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:08.368177 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:08.368174    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5q942\" (UniqueName: \"kubernetes.io/projected/113abc93-9162-4924-a232-29bceb7ef71d-kube-api-access-5q942\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:08.964705 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:08.964677    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-343fe207-chkvn"
Oct 30 21:58:08.964705 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:08.964702    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-343fe207-chkvn" event={"ID":"113abc93-9162-4924-a232-29bceb7ef71d","Type":"ContainerDied","Data":"45fe2674d21c942721e2d9b4a3eacf097f54bb7669d4846f7f3c4eaa74cba488"}
Oct 30 21:58:08.964963 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:08.964747    5061 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="45fe2674d21c942721e2d9b4a3eacf097f54bb7669d4846f7f3c4eaa74cba488"
Oct 30 21:58:12.787567 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:12.787519    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-5be0d1/distinct-w2-5be0d1-kserve-6b85698c8-sxn9k"]
Oct 30 21:58:12.788269 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:12.788244    5061 cpu_manager.go:414] "containerMap: removing container" podUID="113abc93-9162-4924-a232-29bceb7ef71d" containerName="revoke-keys"
Oct 30 21:58:12.788319 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:12.788271    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="113abc93-9162-4924-a232-29bceb7ef71d" containerName="revoke-keys"
Oct 30 21:58:12.788400 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:12.788389    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="113abc93-9162-4924-a232-29bceb7ef71d" containerName="revoke-keys"
Oct 30 21:58:12.792660 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:12.792642    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-5be0d1/distinct-w2-5be0d1-kserve-6b85698c8-sxn9k"
Oct 30 21:58:12.889640 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:12.889608    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-5be0d1\"/\"distinct-w2-5be0d1-kserve-self-signed-certs\""
Oct 30 21:58:12.894466 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:12.894432    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-5be0d1/distinct-w2-5be0d1-kserve-6b85698c8-sxn9k"]
Oct 30 21:58:12.916036 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:12.916004    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xv4l8\" (UniqueName: \"kubernetes.io/projected/7a2dbfc5-908f-41d1-baff-074c56d2cf16-kube-api-access-xv4l8\") pod \"distinct-w2-5be0d1-kserve-6b85698c8-sxn9k\" (UID: \"7a2dbfc5-908f-41d1-baff-074c56d2cf16\") " pod="e2e-models-e2e-worker-w2-5be0d1/distinct-w2-5be0d1-kserve-6b85698c8-sxn9k"
Oct 30 21:58:12.916167 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:12.916060    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a2dbfc5-908f-41d1-baff-074c56d2cf16-dshm\") pod \"distinct-w2-5be0d1-kserve-6b85698c8-sxn9k\" (UID: \"7a2dbfc5-908f-41d1-baff-074c56d2cf16\") " pod="e2e-models-e2e-worker-w2-5be0d1/distinct-w2-5be0d1-kserve-6b85698c8-sxn9k"
Oct 30 21:58:12.916167 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:12.916149    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a2dbfc5-908f-41d1-baff-074c56d2cf16-tmp-dir\") pod \"distinct-w2-5be0d1-kserve-6b85698c8-sxn9k\" (UID: \"7a2dbfc5-908f-41d1-baff-074c56d2cf16\") " pod="e2e-models-e2e-worker-w2-5be0d1/distinct-w2-5be0d1-kserve-6b85698c8-sxn9k"
Oct 30 21:58:12.916243 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:12.916208    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a2dbfc5-908f-41d1-baff-074c56d2cf16-home\") pod \"distinct-w2-5be0d1-kserve-6b85698c8-sxn9k\" (UID: \"7a2dbfc5-908f-41d1-baff-074c56d2cf16\") " pod="e2e-models-e2e-worker-w2-5be0d1/distinct-w2-5be0d1-kserve-6b85698c8-sxn9k"
Oct 30 21:58:12.916379 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:12.916355    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a2dbfc5-908f-41d1-baff-074c56d2cf16-model-cache\") pod \"distinct-w2-5be0d1-kserve-6b85698c8-sxn9k\" (UID: \"7a2dbfc5-908f-41d1-baff-074c56d2cf16\") " pod="e2e-models-e2e-worker-w2-5be0d1/distinct-w2-5be0d1-kserve-6b85698c8-sxn9k"
Oct 30 21:58:12.916451 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:12.916433    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a2dbfc5-908f-41d1-baff-074c56d2cf16-tls-certs\") pod \"distinct-w2-5be0d1-kserve-6b85698c8-sxn9k\" (UID: \"7a2dbfc5-908f-41d1-baff-074c56d2cf16\") " pod="e2e-models-e2e-worker-w2-5be0d1/distinct-w2-5be0d1-kserve-6b85698c8-sxn9k"
Oct 30 21:58:13.017421 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:13.017385    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a2dbfc5-908f-41d1-baff-074c56d2cf16-model-cache\") pod \"distinct-w2-5be0d1-kserve-6b85698c8-sxn9k\" (UID: \"7a2dbfc5-908f-41d1-baff-074c56d2cf16\") " pod="e2e-models-e2e-worker-w2-5be0d1/distinct-w2-5be0d1-kserve-6b85698c8-sxn9k"
Oct 30 21:58:13.017625 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:13.017435    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a2dbfc5-908f-41d1-baff-074c56d2cf16-tls-certs\") pod \"distinct-w2-5be0d1-kserve-6b85698c8-sxn9k\" (UID: \"7a2dbfc5-908f-41d1-baff-074c56d2cf16\") " pod="e2e-models-e2e-worker-w2-5be0d1/distinct-w2-5be0d1-kserve-6b85698c8-sxn9k"
Oct 30 21:58:13.017625 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:13.017502    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xv4l8\" (UniqueName: \"kubernetes.io/projected/7a2dbfc5-908f-41d1-baff-074c56d2cf16-kube-api-access-xv4l8\") pod \"distinct-w2-5be0d1-kserve-6b85698c8-sxn9k\" (UID: \"7a2dbfc5-908f-41d1-baff-074c56d2cf16\") " pod="e2e-models-e2e-worker-w2-5be0d1/distinct-w2-5be0d1-kserve-6b85698c8-sxn9k"
Oct 30 21:58:13.017746 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:13.017650    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a2dbfc5-908f-41d1-baff-074c56d2cf16-dshm\") pod \"distinct-w2-5be0d1-kserve-6b85698c8-sxn9k\" (UID: \"7a2dbfc5-908f-41d1-baff-074c56d2cf16\") " pod="e2e-models-e2e-worker-w2-5be0d1/distinct-w2-5be0d1-kserve-6b85698c8-sxn9k"
Oct 30 21:58:13.017746 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:13.017695    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a2dbfc5-908f-41d1-baff-074c56d2cf16-tmp-dir\") pod \"distinct-w2-5be0d1-kserve-6b85698c8-sxn9k\" (UID: \"7a2dbfc5-908f-41d1-baff-074c56d2cf16\") " pod="e2e-models-e2e-worker-w2-5be0d1/distinct-w2-5be0d1-kserve-6b85698c8-sxn9k"
Oct 30 21:58:13.017746 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:13.017722    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a2dbfc5-908f-41d1-baff-074c56d2cf16-home\") pod \"distinct-w2-5be0d1-kserve-6b85698c8-sxn9k\" (UID: \"7a2dbfc5-908f-41d1-baff-074c56d2cf16\") " pod="e2e-models-e2e-worker-w2-5be0d1/distinct-w2-5be0d1-kserve-6b85698c8-sxn9k"
Oct 30 21:58:13.017894 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:13.017847    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a2dbfc5-908f-41d1-baff-074c56d2cf16-model-cache\") pod \"distinct-w2-5be0d1-kserve-6b85698c8-sxn9k\" (UID: \"7a2dbfc5-908f-41d1-baff-074c56d2cf16\") " pod="e2e-models-e2e-worker-w2-5be0d1/distinct-w2-5be0d1-kserve-6b85698c8-sxn9k"
Oct 30 21:58:13.018021 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:13.018000    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a2dbfc5-908f-41d1-baff-074c56d2cf16-home\") pod \"distinct-w2-5be0d1-kserve-6b85698c8-sxn9k\" (UID: \"7a2dbfc5-908f-41d1-baff-074c56d2cf16\") " pod="e2e-models-e2e-worker-w2-5be0d1/distinct-w2-5be0d1-kserve-6b85698c8-sxn9k"
Oct 30 21:58:13.018251 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:13.018224    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a2dbfc5-908f-41d1-baff-074c56d2cf16-tmp-dir\") pod \"distinct-w2-5be0d1-kserve-6b85698c8-sxn9k\" (UID: \"7a2dbfc5-908f-41d1-baff-074c56d2cf16\") " pod="e2e-models-e2e-worker-w2-5be0d1/distinct-w2-5be0d1-kserve-6b85698c8-sxn9k"
Oct 30 21:58:13.019938 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:13.019916    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a2dbfc5-908f-41d1-baff-074c56d2cf16-dshm\") pod \"distinct-w2-5be0d1-kserve-6b85698c8-sxn9k\" (UID: \"7a2dbfc5-908f-41d1-baff-074c56d2cf16\") " pod="e2e-models-e2e-worker-w2-5be0d1/distinct-w2-5be0d1-kserve-6b85698c8-sxn9k"
Oct 30 21:58:13.020030 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:13.020019    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a2dbfc5-908f-41d1-baff-074c56d2cf16-tls-certs\") pod \"distinct-w2-5be0d1-kserve-6b85698c8-sxn9k\" (UID: \"7a2dbfc5-908f-41d1-baff-074c56d2cf16\") " pod="e2e-models-e2e-worker-w2-5be0d1/distinct-w2-5be0d1-kserve-6b85698c8-sxn9k"
Oct 30 21:58:13.071739 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:13.071715    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xv4l8\" (UniqueName: \"kubernetes.io/projected/7a2dbfc5-908f-41d1-baff-074c56d2cf16-kube-api-access-xv4l8\") pod \"distinct-w2-5be0d1-kserve-6b85698c8-sxn9k\" (UID: \"7a2dbfc5-908f-41d1-baff-074c56d2cf16\") " pod="e2e-models-e2e-worker-w2-5be0d1/distinct-w2-5be0d1-kserve-6b85698c8-sxn9k"
Oct 30 21:58:13.104701 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:13.104672    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-5be0d1/distinct-w2-5be0d1-kserve-6b85698c8-sxn9k"
Oct 30 21:58:13.515330 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:13.515220    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-5be0d1/distinct-w2-5be0d1-kserve-6b85698c8-sxn9k"]
Oct 30 21:58:13.520304 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:58:13.520270    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7a2dbfc5_908f_41d1_baff_074c56d2cf16.slice/crio-3e16833a8d2df226baf3ffea5b9c50c4a1b5be08d2f53f497c7030700ef2d9f8 WatchSource:0}: Error finding container 3e16833a8d2df226baf3ffea5b9c50c4a1b5be08d2f53f497c7030700ef2d9f8: Status 404 returned error can't find the container with id 3e16833a8d2df226baf3ffea5b9c50c4a1b5be08d2f53f497c7030700ef2d9f8
Oct 30 21:58:13.995500 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:13.995464    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-5be0d1/distinct-w2-5be0d1-kserve-6b85698c8-sxn9k" event={"ID":"7a2dbfc5-908f-41d1-baff-074c56d2cf16","Type":"ContainerStarted","Data":"eec9588e46967a83b7f37b412b19ec567120263bae9533e119ae2b46151c015b"}
Oct 30 21:58:13.995500 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:13.995506    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-5be0d1/distinct-w2-5be0d1-kserve-6b85698c8-sxn9k" event={"ID":"7a2dbfc5-908f-41d1-baff-074c56d2cf16","Type":"ContainerStarted","Data":"3e16833a8d2df226baf3ffea5b9c50c4a1b5be08d2f53f497c7030700ef2d9f8"}
Oct 30 21:58:14.104845 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:14.104820    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-5be0d1/distinct-w2-5be0d1-kserve-6b85698c8-sxn9k"
Oct 30 21:58:14.106916 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:14.106872    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-5be0d1/distinct-w2-5be0d1-kserve-6b85698c8-sxn9k" podStartSLOduration=2.106859385 podStartE2EDuration="2.106859385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:14.10478636 +0000 UTC m=+947.717556119" watchObservedRunningTime="2026-10-30 21:58:14.106859385 +0000 UTC m=+947.719629143"
Oct 30 21:58:20.920925 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:20.920884    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4fx5l"]
Oct 30 21:58:20.933796 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:20.933769    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4fx5l"
Oct 30 21:58:20.973757 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:20.968181    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-znswb"]
Oct 30 21:58:21.005632 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:21.005326    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-d6th4\""
Oct 30 21:58:21.041552 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:21.022680    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c55fb717-8e3a-4c50-b0e7-249115db4dda-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-4fx5l\" (UID: \"c55fb717-8e3a-4c50-b0e7-249115db4dda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4fx5l"
Oct 30 21:58:21.041552 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:21.022777    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c55fb717-8e3a-4c50-b0e7-249115db4dda-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-4fx5l\" (UID: \"c55fb717-8e3a-4c50-b0e7-249115db4dda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4fx5l"
Oct 30 21:58:21.041552 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:21.022814    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c55fb717-8e3a-4c50-b0e7-249115db4dda-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-4fx5l\" (UID: \"c55fb717-8e3a-4c50-b0e7-249115db4dda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4fx5l"
Oct 30 21:58:21.041552 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:21.022855    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c55fb717-8e3a-4c50-b0e7-249115db4dda-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-4fx5l\" (UID: \"c55fb717-8e3a-4c50-b0e7-249115db4dda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4fx5l"
Oct 30 21:58:21.041552 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:21.022942    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c55fb717-8e3a-4c50-b0e7-249115db4dda-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-4fx5l\" (UID: \"c55fb717-8e3a-4c50-b0e7-249115db4dda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4fx5l"
Oct 30 21:58:21.041552 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:21.023025    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c55fb717-8e3a-4c50-b0e7-249115db4dda-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-4fx5l\" (UID: \"c55fb717-8e3a-4c50-b0e7-249115db4dda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4fx5l"
Oct 30 21:58:21.041552 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:21.023060    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c55fb717-8e3a-4c50-b0e7-249115db4dda-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-4fx5l\" (UID: \"c55fb717-8e3a-4c50-b0e7-249115db4dda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4fx5l"
Oct 30 21:58:21.041552 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:21.023117    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c55fb717-8e3a-4c50-b0e7-249115db4dda-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-4fx5l\" (UID: \"c55fb717-8e3a-4c50-b0e7-249115db4dda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4fx5l"
Oct 30 21:58:21.041552 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:21.023239    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nf6cj\" (UniqueName: \"kubernetes.io/projected/c55fb717-8e3a-4c50-b0e7-249115db4dda-kube-api-access-nf6cj\") pod \"maas-default-gateway-openshift-default-694db55c87-4fx5l\" (UID: \"c55fb717-8e3a-4c50-b0e7-249115db4dda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4fx5l"
Oct 30 21:58:21.080668 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:21.080637    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-znswb"
Oct 30 21:58:21.206603 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:21.130615    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gg4hd"]
Oct 30 21:58:21.229587 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:21.199088    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8cf2cfc-250e-47a7-a8ef-25f6e343afd8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-znswb\" (UID: \"e8cf2cfc-250e-47a7-a8ef-25f6e343afd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-znswb"
Oct 30 21:58:21.229899 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:21.229863    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8cf2cfc-250e-47a7-a8ef-25f6e343afd8-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-znswb\" (UID: \"e8cf2cfc-250e-47a7-a8ef-25f6e343afd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-znswb"
Oct 30 21:58:21.230167 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:21.230133    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c55fb717-8e3a-4c50-b0e7-249115db4dda-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-4fx5l\" (UID: \"c55fb717-8e3a-4c50-b0e7-249115db4dda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4fx5l"
Oct 30 21:58:21.271553 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:21.256780    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c55fb717-8e3a-4c50-b0e7-249115db4dda-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-4fx5l\" (UID: \"c55fb717-8e3a-4c50-b0e7-249115db4dda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4fx5l"
Oct 30 21:58:21.271553 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:21.256859    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c55fb717-8e3a-4c50-b0e7-249115db4dda-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-4fx5l\" (UID: \"c55fb717-8e3a-4c50-b0e7-249115db4dda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4fx5l"
Oct 30 21:58:21.271553 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:21.256909    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c55fb717-8e3a-4c50-b0e7-249115db4dda-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-4fx5l\" (UID: \"c55fb717-8e3a-4c50-b0e7-249115db4dda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4fx5l"
Oct 30 21:58:21.329552 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:21.269645    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8cf2cfc-250e-47a7-a8ef-25f6e343afd8-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-znswb\" (UID: \"e8cf2cfc-250e-47a7-a8ef-25f6e343afd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-znswb"
Oct 30 21:58:21.329865 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:21.329839    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8cf2cfc-250e-47a7-a8ef-25f6e343afd8-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-znswb\" (UID: \"e8cf2cfc-250e-47a7-a8ef-25f6e343afd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-znswb"
Oct 30 21:58:21.329994 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:21.329978    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8cf2cfc-250e-47a7-a8ef-25f6e343afd8-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-znswb\" (UID: \"e8cf2cfc-250e-47a7-a8ef-25f6e343afd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-znswb"
Oct 30 21:58:21.330133 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:21.330118    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c55fb717-8e3a-4c50-b0e7-249115db4dda-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-4fx5l\" (UID: \"c55fb717-8e3a-4c50-b0e7-249115db4dda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4fx5l"
Oct 30 21:58:21.330329 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:21.330284    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c55fb717-8e3a-4c50-b0e7-249115db4dda-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-4fx5l\" (UID: \"c55fb717-8e3a-4c50-b0e7-249115db4dda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4fx5l"
Oct 30 21:58:21.330480 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:21.330446    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c55fb717-8e3a-4c50-b0e7-249115db4dda-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-4fx5l\" (UID: \"c55fb717-8e3a-4c50-b0e7-249115db4dda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4fx5l"
Oct 30 21:58:21.330635 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:21.330618    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c55fb717-8e3a-4c50-b0e7-249115db4dda-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-4fx5l\" (UID: \"c55fb717-8e3a-4c50-b0e7-249115db4dda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4fx5l"
Oct 30 21:58:21.330765 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:21.330752    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8cf2cfc-250e-47a7-a8ef-25f6e343afd8-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-znswb\" (UID: \"e8cf2cfc-250e-47a7-a8ef-25f6e343afd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-znswb"
Oct 30 21:58:21.330874 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:21.330861    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8cf2cfc-250e-47a7-a8ef-25f6e343afd8-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-znswb\" (UID: \"e8cf2cfc-250e-47a7-a8ef-25f6e343afd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-znswb"
Oct 30 21:58:21.330980 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:21.330967    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8k7f\" (UniqueName: \"kubernetes.io/projected/e8cf2cfc-250e-47a7-a8ef-25f6e343afd8-kube-api-access-s8k7f\") pod \"maas-default-gateway-openshift-default-694db55c87-znswb\" (UID: \"e8cf2cfc-250e-47a7-a8ef-25f6e343afd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-znswb"
Oct 30 21:58:21.331139 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:21.331125    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8cf2cfc-250e-47a7-a8ef-25f6e343afd8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-znswb\" (UID: \"e8cf2cfc-250e-47a7-a8ef-25f6e343afd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-znswb"
Oct 30 21:58:21.331256 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:21.331243    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nf6cj\" (UniqueName: \"kubernetes.io/projected/c55fb717-8e3a-4c50-b0e7-249115db4dda-kube-api-access-nf6cj\") pod \"maas-default-gateway-openshift-default-694db55c87-4fx5l\" (UID: \"c55fb717-8e3a-4c50-b0e7-249115db4dda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4fx5l"
Oct 30 21:58:21.357960 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:21.357908    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4fx5l"]
Oct 30 21:58:21.362934 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:21.362908    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-znswb"]
Oct 30 21:58:21.363142 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:21.363126    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gg4hd"]
Oct 30 21:58:21.363275 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:21.361139    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c55fb717-8e3a-4c50-b0e7-249115db4dda-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-4fx5l\" (UID: \"c55fb717-8e3a-4c50-b0e7-249115db4dda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4fx5l"
Oct 30 21:58:21.363491 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:21.363470    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gg4hd"
Oct 30 21:58:21.365057 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:21.365034    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c55fb717-8e3a-4c50-b0e7-249115db4dda-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-4fx5l\" (UID: \"c55fb717-8e3a-4c50-b0e7-249115db4dda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4fx5l"
Oct 30 21:58:21.407459 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:21.407425    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c55fb717-8e3a-4c50-b0e7-249115db4dda-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-4fx5l\" (UID: \"c55fb717-8e3a-4c50-b0e7-249115db4dda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4fx5l"
Oct 30 21:58:21.407990 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:21.407965    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c55fb717-8e3a-4c50-b0e7-249115db4dda-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-4fx5l\" (UID: \"c55fb717-8e3a-4c50-b0e7-249115db4dda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4fx5l"
Oct 30 21:58:21.436143 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:21.436111    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c55fb717-8e3a-4c50-b0e7-249115db4dda-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-4fx5l\" (UID: \"c55fb717-8e3a-4c50-b0e7-249115db4dda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4fx5l"
Oct 30 21:58:21.509817 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:21.454655    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8cf2cfc-250e-47a7-a8ef-25f6e343afd8-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-znswb\" (UID: \"e8cf2cfc-250e-47a7-a8ef-25f6e343afd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-znswb"
Oct 30 21:58:21.509817 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:21.454817    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8cf2cfc-250e-47a7-a8ef-25f6e343afd8-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-znswb\" (UID: \"e8cf2cfc-250e-47a7-a8ef-25f6e343afd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-znswb"
Oct 30 21:58:21.509817 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:21.454956    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqdp8\" (UniqueName: \"kubernetes.io/projected/fc913c19-cac2-4d4e-93fb-06fddd58ed78-kube-api-access-gqdp8\") pod \"maas-default-gateway-openshift-default-694db55c87-gg4hd\" (UID: \"fc913c19-cac2-4d4e-93fb-06fddd58ed78\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gg4hd"
Oct 30 21:58:21.509817 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:21.455089    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8cf2cfc-250e-47a7-a8ef-25f6e343afd8-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-znswb\" (UID: \"e8cf2cfc-250e-47a7-a8ef-25f6e343afd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-znswb"
Oct 30 21:58:21.509817 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:21.455142    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc913c19-cac2-4d4e-93fb-06fddd58ed78-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-gg4hd\" (UID: \"fc913c19-cac2-4d4e-93fb-06fddd58ed78\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gg4hd"
Oct 30 21:58:21.509817 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:21.455429    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc913c19-cac2-4d4e-93fb-06fddd58ed78-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-gg4hd\" (UID: \"fc913c19-cac2-4d4e-93fb-06fddd58ed78\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gg4hd"
Oct 30 21:58:21.509817 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:21.455563    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8cf2cfc-250e-47a7-a8ef-25f6e343afd8-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-znswb\" (UID: \"e8cf2cfc-250e-47a7-a8ef-25f6e343afd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-znswb"
Oct 30 21:58:21.509817 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:21.455603    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8cf2cfc-250e-47a7-a8ef-25f6e343afd8-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-znswb\" (UID: \"e8cf2cfc-250e-47a7-a8ef-25f6e343afd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-znswb"
Oct 30 21:58:21.509817 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:21.455731    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8k7f\" (UniqueName: \"kubernetes.io/projected/e8cf2cfc-250e-47a7-a8ef-25f6e343afd8-kube-api-access-s8k7f\") pod \"maas-default-gateway-openshift-default-694db55c87-znswb\" (UID: \"e8cf2cfc-250e-47a7-a8ef-25f6e343afd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-znswb"
Oct 30 21:58:21.509817 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:21.455897    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc913c19-cac2-4d4e-93fb-06fddd58ed78-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-gg4hd\" (UID: \"fc913c19-cac2-4d4e-93fb-06fddd58ed78\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gg4hd"
Oct 30 21:58:21.509817 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:21.456027    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8cf2cfc-250e-47a7-a8ef-25f6e343afd8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-znswb\" (UID: \"e8cf2cfc-250e-47a7-a8ef-25f6e343afd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-znswb"
Oct 30 21:58:21.509817 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:21.456058    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc913c19-cac2-4d4e-93fb-06fddd58ed78-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-gg4hd\" (UID: \"fc913c19-cac2-4d4e-93fb-06fddd58ed78\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gg4hd"
Oct 30 21:58:21.509817 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:21.456202    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc913c19-cac2-4d4e-93fb-06fddd58ed78-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-gg4hd\" (UID: \"fc913c19-cac2-4d4e-93fb-06fddd58ed78\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gg4hd"
Oct 30 21:58:21.509817 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:21.456346    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8cf2cfc-250e-47a7-a8ef-25f6e343afd8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-znswb\" (UID: \"e8cf2cfc-250e-47a7-a8ef-25f6e343afd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-znswb"
Oct 30 21:58:21.509817 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:21.456497    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc913c19-cac2-4d4e-93fb-06fddd58ed78-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-gg4hd\" (UID: \"fc913c19-cac2-4d4e-93fb-06fddd58ed78\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gg4hd"
Oct 30 21:58:21.509817 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:21.456636    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc913c19-cac2-4d4e-93fb-06fddd58ed78-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-gg4hd\" (UID: \"fc913c19-cac2-4d4e-93fb-06fddd58ed78\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gg4hd"
Oct 30 21:58:21.509817 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:21.456675    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8cf2cfc-250e-47a7-a8ef-25f6e343afd8-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-znswb\" (UID: \"e8cf2cfc-250e-47a7-a8ef-25f6e343afd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-znswb"
Oct 30 21:58:21.509817 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:21.464455    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc913c19-cac2-4d4e-93fb-06fddd58ed78-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-gg4hd\" (UID: \"fc913c19-cac2-4d4e-93fb-06fddd58ed78\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gg4hd"
Oct 30 21:58:21.509817 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:21.484957    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c55fb717-8e3a-4c50-b0e7-249115db4dda-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-4fx5l\" (UID: \"c55fb717-8e3a-4c50-b0e7-249115db4dda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4fx5l"
Oct 30 21:58:21.509817 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:21.485782    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8cf2cfc-250e-47a7-a8ef-25f6e343afd8-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-znswb\" (UID: \"e8cf2cfc-250e-47a7-a8ef-25f6e343afd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-znswb"
Oct 30 21:58:21.509817 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:21.489136    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8cf2cfc-250e-47a7-a8ef-25f6e343afd8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-znswb\" (UID: \"e8cf2cfc-250e-47a7-a8ef-25f6e343afd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-znswb"
Oct 30 21:58:21.509817 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:21.507720    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8cf2cfc-250e-47a7-a8ef-25f6e343afd8-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-znswb\" (UID: \"e8cf2cfc-250e-47a7-a8ef-25f6e343afd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-znswb"
Oct 30 21:58:21.511173 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:21.511124    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8cf2cfc-250e-47a7-a8ef-25f6e343afd8-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-znswb\" (UID: \"e8cf2cfc-250e-47a7-a8ef-25f6e343afd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-znswb"
Oct 30 21:58:21.511515 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:21.511449    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8cf2cfc-250e-47a7-a8ef-25f6e343afd8-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-znswb\" (UID: \"e8cf2cfc-250e-47a7-a8ef-25f6e343afd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-znswb"
Oct 30 21:58:21.511515 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:21.511467    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c55fb717-8e3a-4c50-b0e7-249115db4dda-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-4fx5l\" (UID: \"c55fb717-8e3a-4c50-b0e7-249115db4dda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4fx5l"
Oct 30 21:58:21.589311 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:21.589266    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc913c19-cac2-4d4e-93fb-06fddd58ed78-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-gg4hd\" (UID: \"fc913c19-cac2-4d4e-93fb-06fddd58ed78\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gg4hd"
Oct 30 21:58:21.590008 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:21.589986    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc913c19-cac2-4d4e-93fb-06fddd58ed78-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-gg4hd\" (UID: \"fc913c19-cac2-4d4e-93fb-06fddd58ed78\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gg4hd"
Oct 30 21:58:21.600713 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:21.600677    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc913c19-cac2-4d4e-93fb-06fddd58ed78-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-gg4hd\" (UID: \"fc913c19-cac2-4d4e-93fb-06fddd58ed78\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gg4hd"
Oct 30 21:58:21.601972 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:21.601951    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc913c19-cac2-4d4e-93fb-06fddd58ed78-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-gg4hd\" (UID: \"fc913c19-cac2-4d4e-93fb-06fddd58ed78\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gg4hd"
Oct 30 21:58:21.602622 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:21.589926    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc913c19-cac2-4d4e-93fb-06fddd58ed78-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-gg4hd\" (UID: \"fc913c19-cac2-4d4e-93fb-06fddd58ed78\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gg4hd"
Oct 30 21:58:21.603179 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:21.601821    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc913c19-cac2-4d4e-93fb-06fddd58ed78-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-gg4hd\" (UID: \"fc913c19-cac2-4d4e-93fb-06fddd58ed78\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gg4hd"
Oct 30 21:58:21.603339 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:21.590912    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc913c19-cac2-4d4e-93fb-06fddd58ed78-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-gg4hd\" (UID: \"fc913c19-cac2-4d4e-93fb-06fddd58ed78\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gg4hd"
Oct 30 21:58:21.603477 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:21.602501    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc913c19-cac2-4d4e-93fb-06fddd58ed78-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-gg4hd\" (UID: \"fc913c19-cac2-4d4e-93fb-06fddd58ed78\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gg4hd"
Oct 30 21:58:21.603625 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:21.592009    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8k7f\" (UniqueName: \"kubernetes.io/projected/e8cf2cfc-250e-47a7-a8ef-25f6e343afd8-kube-api-access-s8k7f\") pod \"maas-default-gateway-openshift-default-694db55c87-znswb\" (UID: \"e8cf2cfc-250e-47a7-a8ef-25f6e343afd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-znswb"
Oct 30 21:58:21.603782 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:21.603758    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc913c19-cac2-4d4e-93fb-06fddd58ed78-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-gg4hd\" (UID: \"fc913c19-cac2-4d4e-93fb-06fddd58ed78\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gg4hd"
Oct 30 21:58:21.603989 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:21.603971    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc913c19-cac2-4d4e-93fb-06fddd58ed78-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-gg4hd\" (UID: \"fc913c19-cac2-4d4e-93fb-06fddd58ed78\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gg4hd"
Oct 30 21:58:21.610137 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:21.610112    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqdp8\" (UniqueName: \"kubernetes.io/projected/fc913c19-cac2-4d4e-93fb-06fddd58ed78-kube-api-access-gqdp8\") pod \"maas-default-gateway-openshift-default-694db55c87-gg4hd\" (UID: \"fc913c19-cac2-4d4e-93fb-06fddd58ed78\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gg4hd"
Oct 30 21:58:21.610733 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:21.610711    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc913c19-cac2-4d4e-93fb-06fddd58ed78-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-gg4hd\" (UID: \"fc913c19-cac2-4d4e-93fb-06fddd58ed78\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gg4hd"
Oct 30 21:58:21.612335 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:21.612282    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc913c19-cac2-4d4e-93fb-06fddd58ed78-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-gg4hd\" (UID: \"fc913c19-cac2-4d4e-93fb-06fddd58ed78\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gg4hd"
Oct 30 21:58:21.613476 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:21.613457    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc913c19-cac2-4d4e-93fb-06fddd58ed78-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-gg4hd\" (UID: \"fc913c19-cac2-4d4e-93fb-06fddd58ed78\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gg4hd"
Oct 30 21:58:21.617992 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:21.617969    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc913c19-cac2-4d4e-93fb-06fddd58ed78-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-gg4hd\" (UID: \"fc913c19-cac2-4d4e-93fb-06fddd58ed78\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gg4hd"
Oct 30 21:58:21.630299 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:21.630216    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8cf2cfc-250e-47a7-a8ef-25f6e343afd8-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-znswb\" (UID: \"e8cf2cfc-250e-47a7-a8ef-25f6e343afd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-znswb"
Oct 30 21:58:21.631556 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:21.630736    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8cf2cfc-250e-47a7-a8ef-25f6e343afd8-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-znswb\" (UID: \"e8cf2cfc-250e-47a7-a8ef-25f6e343afd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-znswb"
Oct 30 21:58:21.631556 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:21.631204    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8cf2cfc-250e-47a7-a8ef-25f6e343afd8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-znswb\" (UID: \"e8cf2cfc-250e-47a7-a8ef-25f6e343afd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-znswb"
Oct 30 21:58:21.631792 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:21.631745    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc913c19-cac2-4d4e-93fb-06fddd58ed78-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-gg4hd\" (UID: \"fc913c19-cac2-4d4e-93fb-06fddd58ed78\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gg4hd"
Oct 30 21:58:21.632280 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:21.632226    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c55fb717-8e3a-4c50-b0e7-249115db4dda-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-4fx5l\" (UID: \"c55fb717-8e3a-4c50-b0e7-249115db4dda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4fx5l"
Oct 30 21:58:21.633286 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:21.633200    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nf6cj\" (UniqueName: \"kubernetes.io/projected/c55fb717-8e3a-4c50-b0e7-249115db4dda-kube-api-access-nf6cj\") pod \"maas-default-gateway-openshift-default-694db55c87-4fx5l\" (UID: \"c55fb717-8e3a-4c50-b0e7-249115db4dda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4fx5l"
Oct 30 21:58:21.697367 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:21.695785    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-znswb"
Oct 30 21:58:21.738749 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:21.724475    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc913c19-cac2-4d4e-93fb-06fddd58ed78-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-gg4hd\" (UID: \"fc913c19-cac2-4d4e-93fb-06fddd58ed78\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gg4hd"
Oct 30 21:58:21.739738 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:21.739671    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqdp8\" (UniqueName: \"kubernetes.io/projected/fc913c19-cac2-4d4e-93fb-06fddd58ed78-kube-api-access-gqdp8\") pod \"maas-default-gateway-openshift-default-694db55c87-gg4hd\" (UID: \"fc913c19-cac2-4d4e-93fb-06fddd58ed78\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gg4hd"
Oct 30 21:58:21.800580 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:21.800546    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4fx5l"
Oct 30 21:58:21.850559 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:21.850256    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gg4hd"
Oct 30 21:58:22.323478 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:58:22.323442    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8cf2cfc_250e_47a7_a8ef_25f6e343afd8.slice/crio-bac251b3c902469a0f1c8fa914a4fb2a11edd9c35489095413aeec6589e83237 WatchSource:0}: Error finding container bac251b3c902469a0f1c8fa914a4fb2a11edd9c35489095413aeec6589e83237: Status 404 returned error can't find the container with id bac251b3c902469a0f1c8fa914a4fb2a11edd9c35489095413aeec6589e83237
Oct 30 21:58:22.324592 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:22.324564    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-znswb"]
Oct 30 21:58:22.327209 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:22.326685    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:58:22.327209 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:22.326793    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:58:22.327209 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:22.326834    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:58:22.330190 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:22.330135    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4fx5l"]
Oct 30 21:58:22.334737 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:22.334712    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gg4hd"]
Oct 30 21:58:22.364683 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:58:22.364637    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc55fb717_8e3a_4c50_b0e7_249115db4dda.slice/crio-202f1e6257937f0892d23c56faf4f05726c55ece844f181d9c6b7c1d88440e24 WatchSource:0}: Error finding container 202f1e6257937f0892d23c56faf4f05726c55ece844f181d9c6b7c1d88440e24: Status 404 returned error can't find the container with id 202f1e6257937f0892d23c56faf4f05726c55ece844f181d9c6b7c1d88440e24
Oct 30 21:58:22.368016 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:22.367977    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:58:22.368112 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:22.368063    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:58:22.368171 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:22.368110    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:58:22.379710 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:22.379682    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:58:22.379817 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:22.379742    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:58:22.379817 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:22.379768    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:58:23.106846 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:23.106813    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-5be0d1/distinct-w2-5be0d1-kserve-6b85698c8-sxn9k"
Oct 30 21:58:23.197545 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:23.197173    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gg4hd" event={"ID":"fc913c19-cac2-4d4e-93fb-06fddd58ed78","Type":"ContainerStarted","Data":"866afe08bd97e1bcfcdf42a15293ecc14d98de6c1951e2a2a30d7fe24a97bccb"}
Oct 30 21:58:23.197707 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:23.197235    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gg4hd" event={"ID":"fc913c19-cac2-4d4e-93fb-06fddd58ed78","Type":"ContainerStarted","Data":"c5bb69d6a6c633c9dba4e801610dc417e891df369010884dca2bafee63f40523"}
Oct 30 21:58:23.199863 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:23.199206    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-znswb" event={"ID":"e8cf2cfc-250e-47a7-a8ef-25f6e343afd8","Type":"ContainerStarted","Data":"409f68272498f576ec2ea40a91a3f64d337a7cee24bcd68b568911882de9d529"}
Oct 30 21:58:23.199863 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:23.199251    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-znswb" event={"ID":"e8cf2cfc-250e-47a7-a8ef-25f6e343afd8","Type":"ContainerStarted","Data":"bac251b3c902469a0f1c8fa914a4fb2a11edd9c35489095413aeec6589e83237"}
Oct 30 21:58:23.205977 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:23.205891    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-5be0d1/distinct-w2-5be0d1-kserve-6b85698c8-sxn9k"
Oct 30 21:58:23.224556 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:23.207640    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4fx5l" event={"ID":"c55fb717-8e3a-4c50-b0e7-249115db4dda","Type":"ContainerStarted","Data":"ff20afd34720205e27cecdf86435ddb1ba57242f30d0310531184f407f2a3fab"}
Oct 30 21:58:23.224556 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:23.207682    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4fx5l" event={"ID":"c55fb717-8e3a-4c50-b0e7-249115db4dda","Type":"ContainerStarted","Data":"202f1e6257937f0892d23c56faf4f05726c55ece844f181d9c6b7c1d88440e24"}
Oct 30 21:58:23.302858 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:23.302738    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-5be0d1/distinct-w2-5be0d1-kserve-6b85698c8-sxn9k"
Oct 30 21:58:23.359684 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:23.359472    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gg4hd" podStartSLOduration=3.359440672 podStartE2EDuration="3.359440672s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:23.333926355 +0000 UTC m=+956.946696112" watchObservedRunningTime="2026-10-30 21:58:23.359440672 +0000 UTC m=+956.972210434"
Oct 30 21:58:23.485960 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:23.480840    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-znswb" podStartSLOduration=3.480820806 podStartE2EDuration="3.480820806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:23.477980604 +0000 UTC m=+957.090750363" watchObservedRunningTime="2026-10-30 21:58:23.480820806 +0000 UTC m=+957.093590566"
Oct 30 21:58:23.700006 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:23.699917    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-znswb"
Oct 30 21:58:23.700651 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:23.700623    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-znswb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Oct 30 21:58:23.700857 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:23.700823    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-znswb" podUID="e8cf2cfc-250e-47a7-a8ef-25f6e343afd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Oct 30 21:58:23.791561 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:23.767096    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4fx5l" podStartSLOduration=3.767073651 podStartE2EDuration="3.767073651s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:23.765080519 +0000 UTC m=+957.377850276" watchObservedRunningTime="2026-10-30 21:58:23.767073651 +0000 UTC m=+957.379843410"
Oct 30 21:58:23.804733 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:23.804470    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4fx5l"
Oct 30 21:58:23.805571 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:23.805437    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-4fx5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Oct 30 21:58:23.805571 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:23.805492    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4fx5l" podUID="c55fb717-8e3a-4c50-b0e7-249115db4dda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Oct 30 21:58:23.877445 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:23.877409    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gg4hd"
Oct 30 21:58:23.928553 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:23.909884    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-gg4hd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Oct 30 21:58:23.928553 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:23.909941    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gg4hd" podUID="fc913c19-cac2-4d4e-93fb-06fddd58ed78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Oct 30 21:58:24.697628 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:24.697591    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-znswb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Oct 30 21:58:24.698136 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:24.697647    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-znswb" podUID="e8cf2cfc-250e-47a7-a8ef-25f6e343afd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Oct 30 21:58:24.704643 ip-10-0-130-60 kubenswrapper[5061]: E1030 21:58:24.704603    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 21:58:24.801183 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:24.801152    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-4fx5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Oct 30 21:58:24.801355 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:24.801209    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4fx5l" podUID="c55fb717-8e3a-4c50-b0e7-249115db4dda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Oct 30 21:58:24.862556 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:24.862500    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-gg4hd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Oct 30 21:58:24.862888 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:24.862821    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gg4hd" podUID="fc913c19-cac2-4d4e-93fb-06fddd58ed78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Oct 30 21:58:25.696843 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:25.696808    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-znswb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Oct 30 21:58:25.697026 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:25.696856    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-znswb" podUID="e8cf2cfc-250e-47a7-a8ef-25f6e343afd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Oct 30 21:58:25.801806 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:25.801757    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-4fx5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Oct 30 21:58:25.802189 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:25.801819    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4fx5l" podUID="c55fb717-8e3a-4c50-b0e7-249115db4dda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Oct 30 21:58:25.855066 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:25.854836    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-gg4hd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Oct 30 21:58:25.855066 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:25.854894    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gg4hd" podUID="fc913c19-cac2-4d4e-93fb-06fddd58ed78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Oct 30 21:58:26.696379 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:26.696351    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-znswb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Oct 30 21:58:26.696622 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:26.696598    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-znswb" podUID="e8cf2cfc-250e-47a7-a8ef-25f6e343afd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Oct 30 21:58:26.801551 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:26.801496    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-4fx5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Oct 30 21:58:26.801723 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:26.801573    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4fx5l" podUID="c55fb717-8e3a-4c50-b0e7-249115db4dda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Oct 30 21:58:26.851006 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:26.850971    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-gg4hd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Oct 30 21:58:26.851601 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:26.851571    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gg4hd" podUID="fc913c19-cac2-4d4e-93fb-06fddd58ed78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Oct 30 21:58:27.696615 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:27.696578    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-znswb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Oct 30 21:58:27.696879 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:27.696651    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-znswb" podUID="e8cf2cfc-250e-47a7-a8ef-25f6e343afd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Oct 30 21:58:27.801245 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:27.801210    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-4fx5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Oct 30 21:58:27.801413 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:27.801259    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4fx5l" podUID="c55fb717-8e3a-4c50-b0e7-249115db4dda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Oct 30 21:58:27.851552 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:27.850885    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-gg4hd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Oct 30 21:58:27.851552 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:27.850949    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gg4hd" podUID="fc913c19-cac2-4d4e-93fb-06fddd58ed78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Oct 30 21:58:28.696997 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:28.696958    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-znswb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Oct 30 21:58:28.697172 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:28.697023    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-znswb" podUID="e8cf2cfc-250e-47a7-a8ef-25f6e343afd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Oct 30 21:58:28.800840 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:28.800806    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-4fx5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Oct 30 21:58:28.801014 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:28.800868    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4fx5l" podUID="c55fb717-8e3a-4c50-b0e7-249115db4dda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Oct 30 21:58:28.850813 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:28.850764    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-gg4hd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Oct 30 21:58:28.851005 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:28.850829    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gg4hd" podUID="fc913c19-cac2-4d4e-93fb-06fddd58ed78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Oct 30 21:58:29.697230 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:29.697179    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-znswb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Oct 30 21:58:29.697758 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:29.697254    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-znswb" podUID="e8cf2cfc-250e-47a7-a8ef-25f6e343afd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Oct 30 21:58:29.801096 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:29.801055    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-4fx5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Oct 30 21:58:29.801263 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:29.801120    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4fx5l" podUID="c55fb717-8e3a-4c50-b0e7-249115db4dda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Oct 30 21:58:29.851329 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:29.851289    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-gg4hd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Oct 30 21:58:29.851576 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:29.851353    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gg4hd" podUID="fc913c19-cac2-4d4e-93fb-06fddd58ed78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Oct 30 21:58:31.434913 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:31.434874    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4fx5l"
Oct 30 21:58:31.435747 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:31.435723    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4fx5l"
Oct 30 21:58:31.436342 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:31.436324    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4fx5l"
Oct 30 21:58:31.522899 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:31.522870    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-znswb"
Oct 30 21:58:31.523263 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:31.523227    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-znswb"
Oct 30 21:58:31.524270 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:31.524249    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-znswb"
Oct 30 21:58:31.527560 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:31.527536    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gg4hd"
Oct 30 21:58:31.527834 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:31.527807    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gg4hd"
Oct 30 21:58:31.528747 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:31.528731    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gg4hd"
Oct 30 21:58:38.011500 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:38.011468    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rsj7x"]
Oct 30 21:58:38.017067 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:38.017047    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rsj7x"
Oct 30 21:58:38.068777 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:38.068754    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-5be0d1-openshift-default-dockercfg-f2nmw\""
Oct 30 21:58:38.069119 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:38.069098    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srg75\" (UniqueName: \"kubernetes.io/projected/fa405318-3b68-43f6-bfe4-d1fc80834e11-kube-api-access-srg75\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rsj7x\" (UID: \"fa405318-3b68-43f6-bfe4-d1fc80834e11\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rsj7x"
Oct 30 21:58:38.069184 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:38.069132    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa405318-3b68-43f6-bfe4-d1fc80834e11-istio-data\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rsj7x\" (UID: \"fa405318-3b68-43f6-bfe4-d1fc80834e11\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rsj7x"
Oct 30 21:58:38.069184 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:38.069159    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa405318-3b68-43f6-bfe4-d1fc80834e11-credential-socket\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rsj7x\" (UID: \"fa405318-3b68-43f6-bfe4-d1fc80834e11\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rsj7x"
Oct 30 21:58:38.069289 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:38.069223    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa405318-3b68-43f6-bfe4-d1fc80834e11-workload-socket\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rsj7x\" (UID: \"fa405318-3b68-43f6-bfe4-d1fc80834e11\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rsj7x"
Oct 30 21:58:38.069354 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:38.069313    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa405318-3b68-43f6-bfe4-d1fc80834e11-istio-podinfo\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rsj7x\" (UID: \"fa405318-3b68-43f6-bfe4-d1fc80834e11\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rsj7x"
Oct 30 21:58:38.069411 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:38.069401    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa405318-3b68-43f6-bfe4-d1fc80834e11-istiod-ca-cert\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rsj7x\" (UID: \"fa405318-3b68-43f6-bfe4-d1fc80834e11\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rsj7x"
Oct 30 21:58:38.069453 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:38.069425    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa405318-3b68-43f6-bfe4-d1fc80834e11-istio-token\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rsj7x\" (UID: \"fa405318-3b68-43f6-bfe4-d1fc80834e11\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rsj7x"
Oct 30 21:58:38.069507 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:38.069457    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa405318-3b68-43f6-bfe4-d1fc80834e11-workload-certs\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rsj7x\" (UID: \"fa405318-3b68-43f6-bfe4-d1fc80834e11\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rsj7x"
Oct 30 21:58:38.069600 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:38.069565    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa405318-3b68-43f6-bfe4-d1fc80834e11-istio-envoy\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rsj7x\" (UID: \"fa405318-3b68-43f6-bfe4-d1fc80834e11\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rsj7x"
Oct 30 21:58:38.120876 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:38.120847    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rsj7x"]
Oct 30 21:58:38.170848 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:38.170820    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa405318-3b68-43f6-bfe4-d1fc80834e11-istiod-ca-cert\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rsj7x\" (UID: \"fa405318-3b68-43f6-bfe4-d1fc80834e11\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rsj7x"
Oct 30 21:58:38.170983 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:38.170869    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa405318-3b68-43f6-bfe4-d1fc80834e11-istio-token\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rsj7x\" (UID: \"fa405318-3b68-43f6-bfe4-d1fc80834e11\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rsj7x"
Oct 30 21:58:38.170983 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:38.170921    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa405318-3b68-43f6-bfe4-d1fc80834e11-workload-certs\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rsj7x\" (UID: \"fa405318-3b68-43f6-bfe4-d1fc80834e11\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rsj7x"
Oct 30 21:58:38.171105 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:38.171079    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa405318-3b68-43f6-bfe4-d1fc80834e11-istio-envoy\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rsj7x\" (UID: \"fa405318-3b68-43f6-bfe4-d1fc80834e11\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rsj7x"
Oct 30 21:58:38.171265 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:38.171244    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srg75\" (UniqueName: \"kubernetes.io/projected/fa405318-3b68-43f6-bfe4-d1fc80834e11-kube-api-access-srg75\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rsj7x\" (UID: \"fa405318-3b68-43f6-bfe4-d1fc80834e11\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rsj7x"
Oct 30 21:58:38.171368 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:38.171288    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa405318-3b68-43f6-bfe4-d1fc80834e11-istio-data\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rsj7x\" (UID: \"fa405318-3b68-43f6-bfe4-d1fc80834e11\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rsj7x"
Oct 30 21:58:38.171368 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:38.171338    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa405318-3b68-43f6-bfe4-d1fc80834e11-credential-socket\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rsj7x\" (UID: \"fa405318-3b68-43f6-bfe4-d1fc80834e11\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rsj7x"
Oct 30 21:58:38.171368 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:38.171345    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa405318-3b68-43f6-bfe4-d1fc80834e11-workload-certs\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rsj7x\" (UID: \"fa405318-3b68-43f6-bfe4-d1fc80834e11\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rsj7x"
Oct 30 21:58:38.171580 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:38.171381    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa405318-3b68-43f6-bfe4-d1fc80834e11-workload-socket\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rsj7x\" (UID: \"fa405318-3b68-43f6-bfe4-d1fc80834e11\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rsj7x"
Oct 30 21:58:38.171580 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:38.171444    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa405318-3b68-43f6-bfe4-d1fc80834e11-istio-podinfo\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rsj7x\" (UID: \"fa405318-3b68-43f6-bfe4-d1fc80834e11\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rsj7x"
Oct 30 21:58:38.171698 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:38.171656    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa405318-3b68-43f6-bfe4-d1fc80834e11-istio-data\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rsj7x\" (UID: \"fa405318-3b68-43f6-bfe4-d1fc80834e11\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rsj7x"
Oct 30 21:58:38.171698 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:38.171659    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa405318-3b68-43f6-bfe4-d1fc80834e11-istiod-ca-cert\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rsj7x\" (UID: \"fa405318-3b68-43f6-bfe4-d1fc80834e11\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rsj7x"
Oct 30 21:58:38.171798 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:38.171778    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa405318-3b68-43f6-bfe4-d1fc80834e11-credential-socket\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rsj7x\" (UID: \"fa405318-3b68-43f6-bfe4-d1fc80834e11\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rsj7x"
Oct 30 21:58:38.172029 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:38.172006    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa405318-3b68-43f6-bfe4-d1fc80834e11-workload-socket\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rsj7x\" (UID: \"fa405318-3b68-43f6-bfe4-d1fc80834e11\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rsj7x"
Oct 30 21:58:38.173371 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:38.173349    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa405318-3b68-43f6-bfe4-d1fc80834e11-istio-envoy\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rsj7x\" (UID: \"fa405318-3b68-43f6-bfe4-d1fc80834e11\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rsj7x"
Oct 30 21:58:38.173799 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:38.173780    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa405318-3b68-43f6-bfe4-d1fc80834e11-istio-podinfo\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rsj7x\" (UID: \"fa405318-3b68-43f6-bfe4-d1fc80834e11\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rsj7x"
Oct 30 21:58:38.225538 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:38.225508    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa405318-3b68-43f6-bfe4-d1fc80834e11-istio-token\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rsj7x\" (UID: \"fa405318-3b68-43f6-bfe4-d1fc80834e11\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rsj7x"
Oct 30 21:58:38.225645 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:38.225626    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srg75\" (UniqueName: \"kubernetes.io/projected/fa405318-3b68-43f6-bfe4-d1fc80834e11-kube-api-access-srg75\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rsj7x\" (UID: \"fa405318-3b68-43f6-bfe4-d1fc80834e11\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rsj7x"
Oct 30 21:58:38.330539 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:38.330499    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rsj7x"
Oct 30 21:58:38.758623 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:38.758497    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rsj7x"]
Oct 30 21:58:38.800505 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:58:38.800479    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa405318_3b68_43f6_bfe4_d1fc80834e11.slice/crio-fd2e5aec210805e68cc57d52a909d3fb37495c583877df4a5e3e84b3d00cf0c4 WatchSource:0}: Error finding container fd2e5aec210805e68cc57d52a909d3fb37495c583877df4a5e3e84b3d00cf0c4: Status 404 returned error can't find the container with id fd2e5aec210805e68cc57d52a909d3fb37495c583877df4a5e3e84b3d00cf0c4
Oct 30 21:58:38.802920 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:38.802887    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:58:38.803004 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:38.802955    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:58:38.803004 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:38.802986    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:58:39.448489 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:39.448448    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rsj7x" event={"ID":"fa405318-3b68-43f6-bfe4-d1fc80834e11","Type":"ContainerStarted","Data":"2baa7f9fab031ecca26095d5131691cc319d4096201108d71660d90906783439"}
Oct 30 21:58:39.448945 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:39.448556    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rsj7x" event={"ID":"fa405318-3b68-43f6-bfe4-d1fc80834e11","Type":"ContainerStarted","Data":"fd2e5aec210805e68cc57d52a909d3fb37495c583877df4a5e3e84b3d00cf0c4"}
Oct 30 21:58:39.677914 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:39.677857    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rsj7x" podStartSLOduration=2.677845715 podStartE2EDuration="2.677845715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:39.630084537 +0000 UTC m=+973.242854322" watchObservedRunningTime="2026-10-30 21:58:39.677845715 +0000 UTC m=+973.290615472"
Oct 30 21:58:40.332554 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:40.332261    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rsj7x"
Oct 30 21:58:40.332554 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:40.332399    5061 patch_prober.go:29] interesting pod/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rsj7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Oct 30 21:58:40.332554 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:40.332442    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rsj7x" podUID="fa405318-3b68-43f6-bfe4-d1fc80834e11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Oct 30 21:58:41.332944 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:41.332904    5061 patch_prober.go:29] interesting pod/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rsj7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Oct 30 21:58:41.333448 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:41.332969    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rsj7x" podUID="fa405318-3b68-43f6-bfe4-d1fc80834e11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Oct 30 21:58:42.332868 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:42.332833    5061 patch_prober.go:29] interesting pod/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rsj7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Oct 30 21:58:42.333114 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:42.332888    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rsj7x" podUID="fa405318-3b68-43f6-bfe4-d1fc80834e11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Oct 30 21:58:43.331551 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:43.330814    5061 patch_prober.go:29] interesting pod/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rsj7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Oct 30 21:58:43.331551 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:43.330872    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rsj7x" podUID="fa405318-3b68-43f6-bfe4-d1fc80834e11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Oct 30 21:58:44.332904 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:44.332864    5061 patch_prober.go:29] interesting pod/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rsj7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Oct 30 21:58:44.333324 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:44.332921    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rsj7x" podUID="fa405318-3b68-43f6-bfe4-d1fc80834e11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Oct 30 21:58:45.331700 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:45.331664    5061 patch_prober.go:29] interesting pod/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rsj7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Oct 30 21:58:45.331875 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:45.331736    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rsj7x" podUID="fa405318-3b68-43f6-bfe4-d1fc80834e11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Oct 30 21:58:46.335433 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:46.335402    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rsj7x"
Oct 30 21:58:46.335897 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:46.335879    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rsj7x"
Oct 30 21:58:46.336484 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:58:46.336465    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rsj7x"
Oct 30 21:58:54.666301 ip-10-0-130-60 kubenswrapper[5061]: E1030 21:58:54.666258    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 21:59:01.846981 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:01.846948    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-14af5c/distinct-w6-14af5c-kserve-7c7cb9d69c-hljl9"]
Oct 30 21:59:01.852369 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:01.852343    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-14af5c/distinct-w6-14af5c-kserve-7c7cb9d69c-hljl9"
Oct 30 21:59:01.930379 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:01.930340    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f8d9486-fb2f-491a-a8d7-c23c21bd116a-model-cache\") pod \"distinct-w6-14af5c-kserve-7c7cb9d69c-hljl9\" (UID: \"1f8d9486-fb2f-491a-a8d7-c23c21bd116a\") " pod="e2e-models-e2e-worker-w6-14af5c/distinct-w6-14af5c-kserve-7c7cb9d69c-hljl9"
Oct 30 21:59:01.930610 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:01.930460    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f8d9486-fb2f-491a-a8d7-c23c21bd116a-tls-certs\") pod \"distinct-w6-14af5c-kserve-7c7cb9d69c-hljl9\" (UID: \"1f8d9486-fb2f-491a-a8d7-c23c21bd116a\") " pod="e2e-models-e2e-worker-w6-14af5c/distinct-w6-14af5c-kserve-7c7cb9d69c-hljl9"
Oct 30 21:59:01.930610 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:01.930482    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f8d9486-fb2f-491a-a8d7-c23c21bd116a-dshm\") pod \"distinct-w6-14af5c-kserve-7c7cb9d69c-hljl9\" (UID: \"1f8d9486-fb2f-491a-a8d7-c23c21bd116a\") " pod="e2e-models-e2e-worker-w6-14af5c/distinct-w6-14af5c-kserve-7c7cb9d69c-hljl9"
Oct 30 21:59:01.930610 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:01.930504    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f8d9486-fb2f-491a-a8d7-c23c21bd116a-tmp-dir\") pod \"distinct-w6-14af5c-kserve-7c7cb9d69c-hljl9\" (UID: \"1f8d9486-fb2f-491a-a8d7-c23c21bd116a\") " pod="e2e-models-e2e-worker-w6-14af5c/distinct-w6-14af5c-kserve-7c7cb9d69c-hljl9"
Oct 30 21:59:01.930821 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:01.930632    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p68cd\" (UniqueName: \"kubernetes.io/projected/1f8d9486-fb2f-491a-a8d7-c23c21bd116a-kube-api-access-p68cd\") pod \"distinct-w6-14af5c-kserve-7c7cb9d69c-hljl9\" (UID: \"1f8d9486-fb2f-491a-a8d7-c23c21bd116a\") " pod="e2e-models-e2e-worker-w6-14af5c/distinct-w6-14af5c-kserve-7c7cb9d69c-hljl9"
Oct 30 21:59:01.930821 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:01.930697    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f8d9486-fb2f-491a-a8d7-c23c21bd116a-home\") pod \"distinct-w6-14af5c-kserve-7c7cb9d69c-hljl9\" (UID: \"1f8d9486-fb2f-491a-a8d7-c23c21bd116a\") " pod="e2e-models-e2e-worker-w6-14af5c/distinct-w6-14af5c-kserve-7c7cb9d69c-hljl9"
Oct 30 21:59:01.949182 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:01.949149    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-14af5c\"/\"distinct-w6-14af5c-kserve-self-signed-certs\""
Oct 30 21:59:01.952786 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:01.952759    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-14af5c/distinct-w6-14af5c-kserve-7c7cb9d69c-hljl9"]
Oct 30 21:59:02.031966 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:02.031928    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f8d9486-fb2f-491a-a8d7-c23c21bd116a-tls-certs\") pod \"distinct-w6-14af5c-kserve-7c7cb9d69c-hljl9\" (UID: \"1f8d9486-fb2f-491a-a8d7-c23c21bd116a\") " pod="e2e-models-e2e-worker-w6-14af5c/distinct-w6-14af5c-kserve-7c7cb9d69c-hljl9"
Oct 30 21:59:02.032177 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:02.031972    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f8d9486-fb2f-491a-a8d7-c23c21bd116a-dshm\") pod \"distinct-w6-14af5c-kserve-7c7cb9d69c-hljl9\" (UID: \"1f8d9486-fb2f-491a-a8d7-c23c21bd116a\") " pod="e2e-models-e2e-worker-w6-14af5c/distinct-w6-14af5c-kserve-7c7cb9d69c-hljl9"
Oct 30 21:59:02.032177 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:02.031999    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f8d9486-fb2f-491a-a8d7-c23c21bd116a-tmp-dir\") pod \"distinct-w6-14af5c-kserve-7c7cb9d69c-hljl9\" (UID: \"1f8d9486-fb2f-491a-a8d7-c23c21bd116a\") " pod="e2e-models-e2e-worker-w6-14af5c/distinct-w6-14af5c-kserve-7c7cb9d69c-hljl9"
Oct 30 21:59:02.032177 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:02.032062    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p68cd\" (UniqueName: \"kubernetes.io/projected/1f8d9486-fb2f-491a-a8d7-c23c21bd116a-kube-api-access-p68cd\") pod \"distinct-w6-14af5c-kserve-7c7cb9d69c-hljl9\" (UID: \"1f8d9486-fb2f-491a-a8d7-c23c21bd116a\") " pod="e2e-models-e2e-worker-w6-14af5c/distinct-w6-14af5c-kserve-7c7cb9d69c-hljl9"
Oct 30 21:59:02.032177 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:02.032130    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f8d9486-fb2f-491a-a8d7-c23c21bd116a-home\") pod \"distinct-w6-14af5c-kserve-7c7cb9d69c-hljl9\" (UID: \"1f8d9486-fb2f-491a-a8d7-c23c21bd116a\") " pod="e2e-models-e2e-worker-w6-14af5c/distinct-w6-14af5c-kserve-7c7cb9d69c-hljl9"
Oct 30 21:59:02.032472 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:02.032449    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f8d9486-fb2f-491a-a8d7-c23c21bd116a-model-cache\") pod \"distinct-w6-14af5c-kserve-7c7cb9d69c-hljl9\" (UID: \"1f8d9486-fb2f-491a-a8d7-c23c21bd116a\") " pod="e2e-models-e2e-worker-w6-14af5c/distinct-w6-14af5c-kserve-7c7cb9d69c-hljl9"
Oct 30 21:59:02.032558 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:02.032468    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f8d9486-fb2f-491a-a8d7-c23c21bd116a-tmp-dir\") pod \"distinct-w6-14af5c-kserve-7c7cb9d69c-hljl9\" (UID: \"1f8d9486-fb2f-491a-a8d7-c23c21bd116a\") " pod="e2e-models-e2e-worker-w6-14af5c/distinct-w6-14af5c-kserve-7c7cb9d69c-hljl9"
Oct 30 21:59:02.032621 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:02.032569    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f8d9486-fb2f-491a-a8d7-c23c21bd116a-home\") pod \"distinct-w6-14af5c-kserve-7c7cb9d69c-hljl9\" (UID: \"1f8d9486-fb2f-491a-a8d7-c23c21bd116a\") " pod="e2e-models-e2e-worker-w6-14af5c/distinct-w6-14af5c-kserve-7c7cb9d69c-hljl9"
Oct 30 21:59:02.032880 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:02.032844    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f8d9486-fb2f-491a-a8d7-c23c21bd116a-model-cache\") pod \"distinct-w6-14af5c-kserve-7c7cb9d69c-hljl9\" (UID: \"1f8d9486-fb2f-491a-a8d7-c23c21bd116a\") " pod="e2e-models-e2e-worker-w6-14af5c/distinct-w6-14af5c-kserve-7c7cb9d69c-hljl9"
Oct 30 21:59:02.034999 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:02.034970    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f8d9486-fb2f-491a-a8d7-c23c21bd116a-dshm\") pod \"distinct-w6-14af5c-kserve-7c7cb9d69c-hljl9\" (UID: \"1f8d9486-fb2f-491a-a8d7-c23c21bd116a\") " pod="e2e-models-e2e-worker-w6-14af5c/distinct-w6-14af5c-kserve-7c7cb9d69c-hljl9"
Oct 30 21:59:02.036138 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:02.036112    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f8d9486-fb2f-491a-a8d7-c23c21bd116a-tls-certs\") pod \"distinct-w6-14af5c-kserve-7c7cb9d69c-hljl9\" (UID: \"1f8d9486-fb2f-491a-a8d7-c23c21bd116a\") " pod="e2e-models-e2e-worker-w6-14af5c/distinct-w6-14af5c-kserve-7c7cb9d69c-hljl9"
Oct 30 21:59:02.089356 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:02.089323    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p68cd\" (UniqueName: \"kubernetes.io/projected/1f8d9486-fb2f-491a-a8d7-c23c21bd116a-kube-api-access-p68cd\") pod \"distinct-w6-14af5c-kserve-7c7cb9d69c-hljl9\" (UID: \"1f8d9486-fb2f-491a-a8d7-c23c21bd116a\") " pod="e2e-models-e2e-worker-w6-14af5c/distinct-w6-14af5c-kserve-7c7cb9d69c-hljl9"
Oct 30 21:59:02.168147 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:02.168066    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-14af5c/distinct-w6-14af5c-kserve-7c7cb9d69c-hljl9"
Oct 30 21:59:02.589414 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:02.589377    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-14af5c/distinct-w6-14af5c-kserve-7c7cb9d69c-hljl9"]
Oct 30 21:59:02.593761 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:59:02.593730    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1f8d9486_fb2f_491a_a8d7_c23c21bd116a.slice/crio-f7779572a148421ec3b4fa477a28a66f3321d09169a0e136211c8b5089979e38 WatchSource:0}: Error finding container f7779572a148421ec3b4fa477a28a66f3321d09169a0e136211c8b5089979e38: Status 404 returned error can't find the container with id f7779572a148421ec3b4fa477a28a66f3321d09169a0e136211c8b5089979e38
Oct 30 21:59:02.659595 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:02.659564    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-14af5c/distinct-w6-14af5c-kserve-7c7cb9d69c-hljl9" event={"ID":"1f8d9486-fb2f-491a-a8d7-c23c21bd116a","Type":"ContainerStarted","Data":"457c372be65206f6c139509e0f4926caeefaf8a0848f0b8831e44587c273745f"}
Oct 30 21:59:02.659729 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:02.659616    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-14af5c/distinct-w6-14af5c-kserve-7c7cb9d69c-hljl9" event={"ID":"1f8d9486-fb2f-491a-a8d7-c23c21bd116a","Type":"ContainerStarted","Data":"f7779572a148421ec3b4fa477a28a66f3321d09169a0e136211c8b5089979e38"}
Oct 30 21:59:02.769571 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:02.769493    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-14af5c/distinct-w6-14af5c-kserve-7c7cb9d69c-hljl9" podStartSLOduration=1.769480436 podStartE2EDuration="1.769480436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:59:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:59:02.768218636 +0000 UTC m=+996.380988404" watchObservedRunningTime="2026-10-30 21:59:02.769480436 +0000 UTC m=+996.382250194"
Oct 30 21:59:03.168746 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:03.168718    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-14af5c/distinct-w6-14af5c-kserve-7c7cb9d69c-hljl9"
Oct 30 21:59:07.538671 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:07.538631    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-bvtk8"]
Oct 30 21:59:07.542835 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:07.542814    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-bvtk8"
Oct 30 21:59:07.637976 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:07.637941    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-bvtk8"]
Oct 30 21:59:07.705059 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:07.705016    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/384f8a66-b2a4-4481-9416-00443b153db9-istio-podinfo\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-bvtk8\" (UID: \"384f8a66-b2a4-4481-9416-00443b153db9\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-bvtk8"
Oct 30 21:59:07.705222 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:07.705118    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/384f8a66-b2a4-4481-9416-00443b153db9-istiod-ca-cert\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-bvtk8\" (UID: \"384f8a66-b2a4-4481-9416-00443b153db9\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-bvtk8"
Oct 30 21:59:07.705222 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:07.705188    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/384f8a66-b2a4-4481-9416-00443b153db9-istio-envoy\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-bvtk8\" (UID: \"384f8a66-b2a4-4481-9416-00443b153db9\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-bvtk8"
Oct 30 21:59:07.705300 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:07.705261    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/384f8a66-b2a4-4481-9416-00443b153db9-istio-data\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-bvtk8\" (UID: \"384f8a66-b2a4-4481-9416-00443b153db9\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-bvtk8"
Oct 30 21:59:07.705300 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:07.705284    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhdkb\" (UniqueName: \"kubernetes.io/projected/384f8a66-b2a4-4481-9416-00443b153db9-kube-api-access-hhdkb\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-bvtk8\" (UID: \"384f8a66-b2a4-4481-9416-00443b153db9\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-bvtk8"
Oct 30 21:59:07.705378 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:07.705355    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/384f8a66-b2a4-4481-9416-00443b153db9-credential-socket\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-bvtk8\" (UID: \"384f8a66-b2a4-4481-9416-00443b153db9\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-bvtk8"
Oct 30 21:59:07.705414 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:07.705401    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/384f8a66-b2a4-4481-9416-00443b153db9-workload-socket\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-bvtk8\" (UID: \"384f8a66-b2a4-4481-9416-00443b153db9\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-bvtk8"
Oct 30 21:59:07.705447 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:07.705417    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/384f8a66-b2a4-4481-9416-00443b153db9-workload-certs\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-bvtk8\" (UID: \"384f8a66-b2a4-4481-9416-00443b153db9\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-bvtk8"
Oct 30 21:59:07.705447 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:07.705440    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/384f8a66-b2a4-4481-9416-00443b153db9-istio-token\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-bvtk8\" (UID: \"384f8a66-b2a4-4481-9416-00443b153db9\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-bvtk8"
Oct 30 21:59:07.806788 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:07.806759    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/384f8a66-b2a4-4481-9416-00443b153db9-istiod-ca-cert\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-bvtk8\" (UID: \"384f8a66-b2a4-4481-9416-00443b153db9\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-bvtk8"
Oct 30 21:59:07.806960 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:07.806813    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/384f8a66-b2a4-4481-9416-00443b153db9-istio-envoy\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-bvtk8\" (UID: \"384f8a66-b2a4-4481-9416-00443b153db9\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-bvtk8"
Oct 30 21:59:07.806960 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:07.806865    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/384f8a66-b2a4-4481-9416-00443b153db9-istio-data\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-bvtk8\" (UID: \"384f8a66-b2a4-4481-9416-00443b153db9\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-bvtk8"
Oct 30 21:59:07.806960 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:07.806886    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhdkb\" (UniqueName: \"kubernetes.io/projected/384f8a66-b2a4-4481-9416-00443b153db9-kube-api-access-hhdkb\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-bvtk8\" (UID: \"384f8a66-b2a4-4481-9416-00443b153db9\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-bvtk8"
Oct 30 21:59:07.806960 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:07.806914    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/384f8a66-b2a4-4481-9416-00443b153db9-credential-socket\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-bvtk8\" (UID: \"384f8a66-b2a4-4481-9416-00443b153db9\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-bvtk8"
Oct 30 21:59:07.806960 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:07.806946    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/384f8a66-b2a4-4481-9416-00443b153db9-workload-socket\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-bvtk8\" (UID: \"384f8a66-b2a4-4481-9416-00443b153db9\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-bvtk8"
Oct 30 21:59:07.807191 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:07.806968    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/384f8a66-b2a4-4481-9416-00443b153db9-workload-certs\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-bvtk8\" (UID: \"384f8a66-b2a4-4481-9416-00443b153db9\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-bvtk8"
Oct 30 21:59:07.807191 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:07.807003    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/384f8a66-b2a4-4481-9416-00443b153db9-istio-token\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-bvtk8\" (UID: \"384f8a66-b2a4-4481-9416-00443b153db9\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-bvtk8"
Oct 30 21:59:07.807191 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:07.807078    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/384f8a66-b2a4-4481-9416-00443b153db9-istio-podinfo\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-bvtk8\" (UID: \"384f8a66-b2a4-4481-9416-00443b153db9\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-bvtk8"
Oct 30 21:59:07.807378 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:07.807350    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/384f8a66-b2a4-4481-9416-00443b153db9-istio-data\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-bvtk8\" (UID: \"384f8a66-b2a4-4481-9416-00443b153db9\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-bvtk8"
Oct 30 21:59:07.807441 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:07.807406    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/384f8a66-b2a4-4481-9416-00443b153db9-workload-certs\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-bvtk8\" (UID: \"384f8a66-b2a4-4481-9416-00443b153db9\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-bvtk8"
Oct 30 21:59:07.807504 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:07.807445    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/384f8a66-b2a4-4481-9416-00443b153db9-credential-socket\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-bvtk8\" (UID: \"384f8a66-b2a4-4481-9416-00443b153db9\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-bvtk8"
Oct 30 21:59:07.807641 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:07.807613    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/384f8a66-b2a4-4481-9416-00443b153db9-workload-socket\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-bvtk8\" (UID: \"384f8a66-b2a4-4481-9416-00443b153db9\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-bvtk8"
Oct 30 21:59:07.807641 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:07.807635    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/384f8a66-b2a4-4481-9416-00443b153db9-istiod-ca-cert\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-bvtk8\" (UID: \"384f8a66-b2a4-4481-9416-00443b153db9\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-bvtk8"
Oct 30 21:59:07.809522 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:07.809505    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/384f8a66-b2a4-4481-9416-00443b153db9-istio-envoy\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-bvtk8\" (UID: \"384f8a66-b2a4-4481-9416-00443b153db9\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-bvtk8"
Oct 30 21:59:07.809598 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:07.809581    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/384f8a66-b2a4-4481-9416-00443b153db9-istio-podinfo\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-bvtk8\" (UID: \"384f8a66-b2a4-4481-9416-00443b153db9\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-bvtk8"
Oct 30 21:59:07.861670 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:07.861635    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhdkb\" (UniqueName: \"kubernetes.io/projected/384f8a66-b2a4-4481-9416-00443b153db9-kube-api-access-hhdkb\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-bvtk8\" (UID: \"384f8a66-b2a4-4481-9416-00443b153db9\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-bvtk8"
Oct 30 21:59:07.861670 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:07.861663    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/384f8a66-b2a4-4481-9416-00443b153db9-istio-token\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-bvtk8\" (UID: \"384f8a66-b2a4-4481-9416-00443b153db9\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-bvtk8"
Oct 30 21:59:08.157160 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:08.157077    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-bvtk8"
Oct 30 21:59:08.585771 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:08.585736    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-bvtk8"]
Oct 30 21:59:08.627221 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:59:08.627193    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod384f8a66_b2a4_4481_9416_00443b153db9.slice/crio-9ba2aa383fdf78a94cb0455b63ec24710cc02e96eaed986f4b5456db5ba20e4f WatchSource:0}: Error finding container 9ba2aa383fdf78a94cb0455b63ec24710cc02e96eaed986f4b5456db5ba20e4f: Status 404 returned error can't find the container with id 9ba2aa383fdf78a94cb0455b63ec24710cc02e96eaed986f4b5456db5ba20e4f
Oct 30 21:59:08.629642 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:08.629610    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:59:08.629732 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:08.629677    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:59:08.629732 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:08.629705    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:59:08.701909 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:08.701876    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-bvtk8" event={"ID":"384f8a66-b2a4-4481-9416-00443b153db9","Type":"ContainerStarted","Data":"f4b6e4efb8ea79279731a9766205fd5a2d69a440b4127d3cc35e63300363e1d4"}
Oct 30 21:59:08.702057 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:08.701931    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-bvtk8" event={"ID":"384f8a66-b2a4-4481-9416-00443b153db9","Type":"ContainerStarted","Data":"9ba2aa383fdf78a94cb0455b63ec24710cc02e96eaed986f4b5456db5ba20e4f"}
Oct 30 21:59:08.818553 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:08.817794    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-bvtk8" podStartSLOduration=1.817774499 podStartE2EDuration="1.817774499s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:59:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:59:08.816432796 +0000 UTC m=+1002.429202573" watchObservedRunningTime="2026-10-30 21:59:08.817774499 +0000 UTC m=+1002.430544257"
Oct 30 21:59:09.158327 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:09.158285    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-bvtk8"
Oct 30 21:59:09.160513 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:09.160485    5061 patch_prober.go:29] interesting pod/e2e-worker-w3-f776cb-openshift-default-74d577586f-bvtk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Oct 30 21:59:09.160653 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:09.160556    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-bvtk8" podUID="384f8a66-b2a4-4481-9416-00443b153db9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Oct 30 21:59:10.157728 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:10.157690    5061 patch_prober.go:29] interesting pod/e2e-worker-w3-f776cb-openshift-default-74d577586f-bvtk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Oct 30 21:59:10.160036 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:10.157768    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-bvtk8" podUID="384f8a66-b2a4-4481-9416-00443b153db9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Oct 30 21:59:11.158290 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:11.158254    5061 patch_prober.go:29] interesting pod/e2e-worker-w3-f776cb-openshift-default-74d577586f-bvtk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Oct 30 21:59:11.158802 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:11.158315    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-bvtk8" podUID="384f8a66-b2a4-4481-9416-00443b153db9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Oct 30 21:59:12.157630 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:12.157595    5061 patch_prober.go:29] interesting pod/e2e-worker-w3-f776cb-openshift-default-74d577586f-bvtk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Oct 30 21:59:12.157804 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:12.157647    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-bvtk8" podUID="384f8a66-b2a4-4481-9416-00443b153db9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Oct 30 21:59:12.168487 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:12.168462    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-14af5c/distinct-w6-14af5c-kserve-7c7cb9d69c-hljl9"
Oct 30 21:59:12.181301 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:12.181275    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-14af5c/distinct-w6-14af5c-kserve-7c7cb9d69c-hljl9"
Oct 30 21:59:12.758827 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:12.758798    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-14af5c/distinct-w6-14af5c-kserve-7c7cb9d69c-hljl9"
Oct 30 21:59:13.157378 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:13.157343    5061 patch_prober.go:29] interesting pod/e2e-worker-w3-f776cb-openshift-default-74d577586f-bvtk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Oct 30 21:59:13.157566 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:13.157402    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-bvtk8" podUID="384f8a66-b2a4-4481-9416-00443b153db9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Oct 30 21:59:14.157818 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:14.157784    5061 patch_prober.go:29] interesting pod/e2e-worker-w3-f776cb-openshift-default-74d577586f-bvtk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Oct 30 21:59:14.158318 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:14.157844    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-bvtk8" podUID="384f8a66-b2a4-4481-9416-00443b153db9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Oct 30 21:59:16.044344 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:16.044309    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-bvtk8"
Oct 30 21:59:16.044995 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:16.044970    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-bvtk8"
Oct 30 21:59:16.046154 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:16.046132    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-bvtk8"
Oct 30 21:59:24.667349 ip-10-0-130-60 kubenswrapper[5061]: E1030 21:59:24.667313    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 21:59:53.358960 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:53.358873    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-17d809/distinct-w4-17d809-kserve-6b8f5c48cd-lhrwp"]
Oct 30 21:59:53.363405 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:53.363380    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-17d809/distinct-w4-17d809-kserve-6b8f5c48cd-lhrwp"
Oct 30 21:59:53.460440 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:53.460399    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-17d809\"/\"distinct-w4-17d809-kserve-self-signed-certs\""
Oct 30 21:59:53.465588 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:53.465560    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-17d809/distinct-w4-17d809-kserve-6b8f5c48cd-lhrwp"]
Oct 30 21:59:53.515073 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:53.515030    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef5e8691-42e1-471b-ba7d-90881028eca7-tls-certs\") pod \"distinct-w4-17d809-kserve-6b8f5c48cd-lhrwp\" (UID: \"ef5e8691-42e1-471b-ba7d-90881028eca7\") " pod="e2e-models-e2e-worker-w4-17d809/distinct-w4-17d809-kserve-6b8f5c48cd-lhrwp"
Oct 30 21:59:53.515282 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:53.515145    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sz84d\" (UniqueName: \"kubernetes.io/projected/ef5e8691-42e1-471b-ba7d-90881028eca7-kube-api-access-sz84d\") pod \"distinct-w4-17d809-kserve-6b8f5c48cd-lhrwp\" (UID: \"ef5e8691-42e1-471b-ba7d-90881028eca7\") " pod="e2e-models-e2e-worker-w4-17d809/distinct-w4-17d809-kserve-6b8f5c48cd-lhrwp"
Oct 30 21:59:53.515282 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:53.515181    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef5e8691-42e1-471b-ba7d-90881028eca7-model-cache\") pod \"distinct-w4-17d809-kserve-6b8f5c48cd-lhrwp\" (UID: \"ef5e8691-42e1-471b-ba7d-90881028eca7\") " pod="e2e-models-e2e-worker-w4-17d809/distinct-w4-17d809-kserve-6b8f5c48cd-lhrwp"
Oct 30 21:59:53.515282 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:53.515224    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef5e8691-42e1-471b-ba7d-90881028eca7-home\") pod \"distinct-w4-17d809-kserve-6b8f5c48cd-lhrwp\" (UID: \"ef5e8691-42e1-471b-ba7d-90881028eca7\") " pod="e2e-models-e2e-worker-w4-17d809/distinct-w4-17d809-kserve-6b8f5c48cd-lhrwp"
Oct 30 21:59:53.515482 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:53.515284    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef5e8691-42e1-471b-ba7d-90881028eca7-tmp-dir\") pod \"distinct-w4-17d809-kserve-6b8f5c48cd-lhrwp\" (UID: \"ef5e8691-42e1-471b-ba7d-90881028eca7\") " pod="e2e-models-e2e-worker-w4-17d809/distinct-w4-17d809-kserve-6b8f5c48cd-lhrwp"
Oct 30 21:59:53.515482 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:53.515426    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef5e8691-42e1-471b-ba7d-90881028eca7-dshm\") pod \"distinct-w4-17d809-kserve-6b8f5c48cd-lhrwp\" (UID: \"ef5e8691-42e1-471b-ba7d-90881028eca7\") " pod="e2e-models-e2e-worker-w4-17d809/distinct-w4-17d809-kserve-6b8f5c48cd-lhrwp"
Oct 30 21:59:53.616631 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:53.616547    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef5e8691-42e1-471b-ba7d-90881028eca7-tls-certs\") pod \"distinct-w4-17d809-kserve-6b8f5c48cd-lhrwp\" (UID: \"ef5e8691-42e1-471b-ba7d-90881028eca7\") " pod="e2e-models-e2e-worker-w4-17d809/distinct-w4-17d809-kserve-6b8f5c48cd-lhrwp"
Oct 30 21:59:53.616631 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:53.616602    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sz84d\" (UniqueName: \"kubernetes.io/projected/ef5e8691-42e1-471b-ba7d-90881028eca7-kube-api-access-sz84d\") pod \"distinct-w4-17d809-kserve-6b8f5c48cd-lhrwp\" (UID: \"ef5e8691-42e1-471b-ba7d-90881028eca7\") " pod="e2e-models-e2e-worker-w4-17d809/distinct-w4-17d809-kserve-6b8f5c48cd-lhrwp"
Oct 30 21:59:53.616631 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:53.616627    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef5e8691-42e1-471b-ba7d-90881028eca7-model-cache\") pod \"distinct-w4-17d809-kserve-6b8f5c48cd-lhrwp\" (UID: \"ef5e8691-42e1-471b-ba7d-90881028eca7\") " pod="e2e-models-e2e-worker-w4-17d809/distinct-w4-17d809-kserve-6b8f5c48cd-lhrwp"
Oct 30 21:59:53.616870 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:53.616655    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef5e8691-42e1-471b-ba7d-90881028eca7-home\") pod \"distinct-w4-17d809-kserve-6b8f5c48cd-lhrwp\" (UID: \"ef5e8691-42e1-471b-ba7d-90881028eca7\") " pod="e2e-models-e2e-worker-w4-17d809/distinct-w4-17d809-kserve-6b8f5c48cd-lhrwp"
Oct 30 21:59:53.616870 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:53.616692    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef5e8691-42e1-471b-ba7d-90881028eca7-tmp-dir\") pod \"distinct-w4-17d809-kserve-6b8f5c48cd-lhrwp\" (UID: \"ef5e8691-42e1-471b-ba7d-90881028eca7\") " pod="e2e-models-e2e-worker-w4-17d809/distinct-w4-17d809-kserve-6b8f5c48cd-lhrwp"
Oct 30 21:59:53.616870 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:53.616737    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef5e8691-42e1-471b-ba7d-90881028eca7-dshm\") pod \"distinct-w4-17d809-kserve-6b8f5c48cd-lhrwp\" (UID: \"ef5e8691-42e1-471b-ba7d-90881028eca7\") " pod="e2e-models-e2e-worker-w4-17d809/distinct-w4-17d809-kserve-6b8f5c48cd-lhrwp"
Oct 30 21:59:53.617089 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:53.617061    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef5e8691-42e1-471b-ba7d-90881028eca7-model-cache\") pod \"distinct-w4-17d809-kserve-6b8f5c48cd-lhrwp\" (UID: \"ef5e8691-42e1-471b-ba7d-90881028eca7\") " pod="e2e-models-e2e-worker-w4-17d809/distinct-w4-17d809-kserve-6b8f5c48cd-lhrwp"
Oct 30 21:59:53.617172 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:53.617106    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef5e8691-42e1-471b-ba7d-90881028eca7-home\") pod \"distinct-w4-17d809-kserve-6b8f5c48cd-lhrwp\" (UID: \"ef5e8691-42e1-471b-ba7d-90881028eca7\") " pod="e2e-models-e2e-worker-w4-17d809/distinct-w4-17d809-kserve-6b8f5c48cd-lhrwp"
Oct 30 21:59:53.617172 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:53.617165    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef5e8691-42e1-471b-ba7d-90881028eca7-tmp-dir\") pod \"distinct-w4-17d809-kserve-6b8f5c48cd-lhrwp\" (UID: \"ef5e8691-42e1-471b-ba7d-90881028eca7\") " pod="e2e-models-e2e-worker-w4-17d809/distinct-w4-17d809-kserve-6b8f5c48cd-lhrwp"
Oct 30 21:59:53.618862 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:53.618841    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef5e8691-42e1-471b-ba7d-90881028eca7-dshm\") pod \"distinct-w4-17d809-kserve-6b8f5c48cd-lhrwp\" (UID: \"ef5e8691-42e1-471b-ba7d-90881028eca7\") " pod="e2e-models-e2e-worker-w4-17d809/distinct-w4-17d809-kserve-6b8f5c48cd-lhrwp"
Oct 30 21:59:53.619179 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:53.619161    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef5e8691-42e1-471b-ba7d-90881028eca7-tls-certs\") pod \"distinct-w4-17d809-kserve-6b8f5c48cd-lhrwp\" (UID: \"ef5e8691-42e1-471b-ba7d-90881028eca7\") " pod="e2e-models-e2e-worker-w4-17d809/distinct-w4-17d809-kserve-6b8f5c48cd-lhrwp"
Oct 30 21:59:53.671875 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:53.671847    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sz84d\" (UniqueName: \"kubernetes.io/projected/ef5e8691-42e1-471b-ba7d-90881028eca7-kube-api-access-sz84d\") pod \"distinct-w4-17d809-kserve-6b8f5c48cd-lhrwp\" (UID: \"ef5e8691-42e1-471b-ba7d-90881028eca7\") " pod="e2e-models-e2e-worker-w4-17d809/distinct-w4-17d809-kserve-6b8f5c48cd-lhrwp"
Oct 30 21:59:53.675681 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:53.675663    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-17d809/distinct-w4-17d809-kserve-6b8f5c48cd-lhrwp"
Oct 30 21:59:54.055054 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:54.055016    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-17d809/distinct-w4-17d809-kserve-6b8f5c48cd-lhrwp"]
Oct 30 21:59:54.059793 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:59:54.059766    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podef5e8691_42e1_471b_ba7d_90881028eca7.slice/crio-0b15259dff7ee7ea86d2e7b698cb27c4e2c76315a7514a64ca19ede8cf051889 WatchSource:0}: Error finding container 0b15259dff7ee7ea86d2e7b698cb27c4e2c76315a7514a64ca19ede8cf051889: Status 404 returned error can't find the container with id 0b15259dff7ee7ea86d2e7b698cb27c4e2c76315a7514a64ca19ede8cf051889
Oct 30 21:59:54.316912 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:54.316822    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-9kcgf"]
Oct 30 21:59:54.321149 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:54.321125    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-9kcgf"
Oct 30 21:59:54.399034 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:54.399000    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-9kcgf"]
Oct 30 21:59:54.430892 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:54.430857    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b521eca-5188-43b6-b589-d0ccab8c4520-istio-envoy\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-9kcgf\" (UID: \"9b521eca-5188-43b6-b589-d0ccab8c4520\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-9kcgf"
Oct 30 21:59:54.431053 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:54.430898    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b521eca-5188-43b6-b589-d0ccab8c4520-workload-socket\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-9kcgf\" (UID: \"9b521eca-5188-43b6-b589-d0ccab8c4520\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-9kcgf"
Oct 30 21:59:54.431143 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:54.431065    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b521eca-5188-43b6-b589-d0ccab8c4520-credential-socket\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-9kcgf\" (UID: \"9b521eca-5188-43b6-b589-d0ccab8c4520\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-9kcgf"
Oct 30 21:59:54.431143 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:54.431125    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b521eca-5188-43b6-b589-d0ccab8c4520-istio-podinfo\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-9kcgf\" (UID: \"9b521eca-5188-43b6-b589-d0ccab8c4520\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-9kcgf"
Oct 30 21:59:54.431256 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:54.431152    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b521eca-5188-43b6-b589-d0ccab8c4520-istiod-ca-cert\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-9kcgf\" (UID: \"9b521eca-5188-43b6-b589-d0ccab8c4520\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-9kcgf"
Oct 30 21:59:54.431256 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:54.431216    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b521eca-5188-43b6-b589-d0ccab8c4520-istio-token\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-9kcgf\" (UID: \"9b521eca-5188-43b6-b589-d0ccab8c4520\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-9kcgf"
Oct 30 21:59:54.431668 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:54.431640    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b521eca-5188-43b6-b589-d0ccab8c4520-workload-certs\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-9kcgf\" (UID: \"9b521eca-5188-43b6-b589-d0ccab8c4520\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-9kcgf"
Oct 30 21:59:54.431810 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:54.431771    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b521eca-5188-43b6-b589-d0ccab8c4520-istio-data\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-9kcgf\" (UID: \"9b521eca-5188-43b6-b589-d0ccab8c4520\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-9kcgf"
Oct 30 21:59:54.431877 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:54.431822    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4qls\" (UniqueName: \"kubernetes.io/projected/9b521eca-5188-43b6-b589-d0ccab8c4520-kube-api-access-g4qls\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-9kcgf\" (UID: \"9b521eca-5188-43b6-b589-d0ccab8c4520\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-9kcgf"
Oct 30 21:59:54.533302 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:54.533265    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b521eca-5188-43b6-b589-d0ccab8c4520-credential-socket\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-9kcgf\" (UID: \"9b521eca-5188-43b6-b589-d0ccab8c4520\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-9kcgf"
Oct 30 21:59:54.533476 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:54.533322    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b521eca-5188-43b6-b589-d0ccab8c4520-istio-podinfo\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-9kcgf\" (UID: \"9b521eca-5188-43b6-b589-d0ccab8c4520\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-9kcgf"
Oct 30 21:59:54.533476 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:54.533348    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b521eca-5188-43b6-b589-d0ccab8c4520-istiod-ca-cert\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-9kcgf\" (UID: \"9b521eca-5188-43b6-b589-d0ccab8c4520\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-9kcgf"
Oct 30 21:59:54.533476 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:54.533402    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b521eca-5188-43b6-b589-d0ccab8c4520-istio-token\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-9kcgf\" (UID: \"9b521eca-5188-43b6-b589-d0ccab8c4520\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-9kcgf"
Oct 30 21:59:54.533476 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:54.533453    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b521eca-5188-43b6-b589-d0ccab8c4520-workload-certs\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-9kcgf\" (UID: \"9b521eca-5188-43b6-b589-d0ccab8c4520\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-9kcgf"
Oct 30 21:59:54.533734 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:54.533582    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b521eca-5188-43b6-b589-d0ccab8c4520-istio-data\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-9kcgf\" (UID: \"9b521eca-5188-43b6-b589-d0ccab8c4520\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-9kcgf"
Oct 30 21:59:54.533734 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:54.533634    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4qls\" (UniqueName: \"kubernetes.io/projected/9b521eca-5188-43b6-b589-d0ccab8c4520-kube-api-access-g4qls\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-9kcgf\" (UID: \"9b521eca-5188-43b6-b589-d0ccab8c4520\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-9kcgf"
Oct 30 21:59:54.533734 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:54.533683    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b521eca-5188-43b6-b589-d0ccab8c4520-istio-envoy\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-9kcgf\" (UID: \"9b521eca-5188-43b6-b589-d0ccab8c4520\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-9kcgf"
Oct 30 21:59:54.533734 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:54.533719    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b521eca-5188-43b6-b589-d0ccab8c4520-workload-socket\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-9kcgf\" (UID: \"9b521eca-5188-43b6-b589-d0ccab8c4520\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-9kcgf"
Oct 30 21:59:54.533924 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:54.533716    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b521eca-5188-43b6-b589-d0ccab8c4520-credential-socket\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-9kcgf\" (UID: \"9b521eca-5188-43b6-b589-d0ccab8c4520\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-9kcgf"
Oct 30 21:59:54.533924 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:54.533837    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b521eca-5188-43b6-b589-d0ccab8c4520-workload-certs\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-9kcgf\" (UID: \"9b521eca-5188-43b6-b589-d0ccab8c4520\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-9kcgf"
Oct 30 21:59:54.533993 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:54.533967    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b521eca-5188-43b6-b589-d0ccab8c4520-istio-data\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-9kcgf\" (UID: \"9b521eca-5188-43b6-b589-d0ccab8c4520\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-9kcgf"
Oct 30 21:59:54.534029 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:54.534012    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b521eca-5188-43b6-b589-d0ccab8c4520-workload-socket\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-9kcgf\" (UID: \"9b521eca-5188-43b6-b589-d0ccab8c4520\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-9kcgf"
Oct 30 21:59:54.534103 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:54.534077    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b521eca-5188-43b6-b589-d0ccab8c4520-istiod-ca-cert\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-9kcgf\" (UID: \"9b521eca-5188-43b6-b589-d0ccab8c4520\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-9kcgf"
Oct 30 21:59:54.535922 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:54.535904    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b521eca-5188-43b6-b589-d0ccab8c4520-istio-envoy\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-9kcgf\" (UID: \"9b521eca-5188-43b6-b589-d0ccab8c4520\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-9kcgf"
Oct 30 21:59:54.536056 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:54.536040    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b521eca-5188-43b6-b589-d0ccab8c4520-istio-podinfo\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-9kcgf\" (UID: \"9b521eca-5188-43b6-b589-d0ccab8c4520\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-9kcgf"
Oct 30 21:59:54.590173 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:54.590105    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b521eca-5188-43b6-b589-d0ccab8c4520-istio-token\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-9kcgf\" (UID: \"9b521eca-5188-43b6-b589-d0ccab8c4520\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-9kcgf"
Oct 30 21:59:54.590321 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:54.590263    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4qls\" (UniqueName: \"kubernetes.io/projected/9b521eca-5188-43b6-b589-d0ccab8c4520-kube-api-access-g4qls\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-9kcgf\" (UID: \"9b521eca-5188-43b6-b589-d0ccab8c4520\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-9kcgf"
Oct 30 21:59:54.634950 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:54.634921    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-9kcgf"
Oct 30 21:59:54.672083 ip-10-0-130-60 kubenswrapper[5061]: E1030 21:59:54.672027    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 21:59:55.058235 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:55.058196    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-17d809/distinct-w4-17d809-kserve-6b8f5c48cd-lhrwp" event={"ID":"ef5e8691-42e1-471b-ba7d-90881028eca7","Type":"ContainerStarted","Data":"a95e9b0f3ae527aeb8d1e8c8f557fc55453a836cd2574420c5a0e220bebd705a"}
Oct 30 21:59:55.058411 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:55.058249    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-17d809/distinct-w4-17d809-kserve-6b8f5c48cd-lhrwp" event={"ID":"ef5e8691-42e1-471b-ba7d-90881028eca7","Type":"ContainerStarted","Data":"0b15259dff7ee7ea86d2e7b698cb27c4e2c76315a7514a64ca19ede8cf051889"}
Oct 30 21:59:55.066602 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:55.066575    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-9kcgf"]
Oct 30 21:59:55.071008 ip-10-0-130-60 kubenswrapper[5061]: W1030 21:59:55.070986    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b521eca_5188_43b6_b589_d0ccab8c4520.slice/crio-1172fe2fb0d1371137d5d596eaa1e54e194e58ab1a82629769b0fa44b913ba51 WatchSource:0}: Error finding container 1172fe2fb0d1371137d5d596eaa1e54e194e58ab1a82629769b0fa44b913ba51: Status 404 returned error can't find the container with id 1172fe2fb0d1371137d5d596eaa1e54e194e58ab1a82629769b0fa44b913ba51
Oct 30 21:59:55.073675 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:55.073642    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:59:55.073752 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:55.073721    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:59:55.073792 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:55.073759    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:59:55.168591 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:55.168546    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-17d809/distinct-w4-17d809-kserve-6b8f5c48cd-lhrwp" podStartSLOduration=2.168515334 podStartE2EDuration="2.168515334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:59:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:59:55.166463011 +0000 UTC m=+1048.779232769" watchObservedRunningTime="2026-10-30 21:59:55.168515334 +0000 UTC m=+1048.781285105"
Oct 30 21:59:55.675900 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:55.675858    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-17d809/distinct-w4-17d809-kserve-6b8f5c48cd-lhrwp"
Oct 30 21:59:56.066249 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:56.066206    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-9kcgf" event={"ID":"9b521eca-5188-43b6-b589-d0ccab8c4520","Type":"ContainerStarted","Data":"bcb2e4028f11ad1fa672f8a687f6cbd4285f037f8856cfeb5e62b899b3f87113"}
Oct 30 21:59:56.066465 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:56.066270    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-9kcgf" event={"ID":"9b521eca-5188-43b6-b589-d0ccab8c4520","Type":"ContainerStarted","Data":"1172fe2fb0d1371137d5d596eaa1e54e194e58ab1a82629769b0fa44b913ba51"}
Oct 30 21:59:56.277623 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:56.277569    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-9kcgf" podStartSLOduration=2.277555237 podStartE2EDuration="2.277555237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:59:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:59:56.274163948 +0000 UTC m=+1049.886933706" watchObservedRunningTime="2026-10-30 21:59:56.277555237 +0000 UTC m=+1049.890324995"
Oct 30 21:59:56.635890 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:56.635851    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-9kcgf"
Oct 30 21:59:56.637730 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:56.637693    5061 patch_prober.go:29] interesting pod/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-9kcgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Oct 30 21:59:56.637868 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:56.637763    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-9kcgf" podUID="9b521eca-5188-43b6-b589-d0ccab8c4520" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Oct 30 21:59:57.636090 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:57.636051    5061 patch_prober.go:29] interesting pod/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-9kcgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Oct 30 21:59:57.636517 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:57.636108    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-9kcgf" podUID="9b521eca-5188-43b6-b589-d0ccab8c4520" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Oct 30 21:59:58.636012 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:58.635976    5061 patch_prober.go:29] interesting pod/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-9kcgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Oct 30 21:59:58.636178 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:58.636031    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-9kcgf" podUID="9b521eca-5188-43b6-b589-d0ccab8c4520" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Oct 30 21:59:59.635680 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:59.635647    5061 patch_prober.go:29] interesting pod/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-9kcgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Oct 30 21:59:59.635998 ip-10-0-130-60 kubenswrapper[5061]: I1030 21:59:59.635705    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-9kcgf" podUID="9b521eca-5188-43b6-b589-d0ccab8c4520" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Oct 30 22:00:00.187172 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:00.187139    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-14af5c-29889960-ww6xn"]
Oct 30 22:00:00.191420 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:00.191398    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5be0d1-29889960-7tvfp"]
Oct 30 22:00:00.191553 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:00.191513    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-14af5c-29889960-ww6xn"
Oct 30 22:00:00.195269 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:00.195250    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5be0d1-29889960-7tvfp"
Oct 30 22:00:00.249170 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:00.242190    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gdmfg\""
Oct 30 22:00:00.249170 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:00.244276    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d70ad2e-86fa-40c7-86e3-98adba4c3913-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-14af5c-29889960-ww6xn\" (UID: \"5d70ad2e-86fa-40c7-86e3-98adba4c3913\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-14af5c-29889960-ww6xn"
Oct 30 22:00:00.249170 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:00.244433    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtjxm\" (UniqueName: \"kubernetes.io/projected/865d5bf1-9788-4c88-8ae6-e1a20f1efa66-kube-api-access-vtjxm\") pod \"maas-api-key-cleanup-e2e-worker-w2-5be0d1-29889960-7tvfp\" (UID: \"865d5bf1-9788-4c88-8ae6-e1a20f1efa66\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5be0d1-29889960-7tvfp"
Oct 30 22:00:00.249170 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:00.244514    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnfc9\" (UniqueName: \"kubernetes.io/projected/5d70ad2e-86fa-40c7-86e3-98adba4c3913-kube-api-access-bnfc9\") pod \"maas-api-key-cleanup-e2e-worker-w6-14af5c-29889960-ww6xn\" (UID: \"5d70ad2e-86fa-40c7-86e3-98adba4c3913\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-14af5c-29889960-ww6xn"
Oct 30 22:00:00.249170 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:00.244563    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/865d5bf1-9788-4c88-8ae6-e1a20f1efa66-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-5be0d1-29889960-7tvfp\" (UID: \"865d5bf1-9788-4c88-8ae6-e1a20f1efa66\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5be0d1-29889960-7tvfp"
Oct 30 22:00:00.249170 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:00.247693    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-17d809-29889960-hwzbr"]
Oct 30 22:00:00.254861 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:00.254838    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-17d809-29889960-hwzbr"
Oct 30 22:00:00.289328 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:00.289298    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-14af5c-29889960-ww6xn"]
Oct 30 22:00:00.295468 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:00.295437    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5be0d1-29889960-7tvfp"]
Oct 30 22:00:00.341601 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:00.341567    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-17d809-29889960-hwzbr"]
Oct 30 22:00:00.345493 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:00.345453    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnfc9\" (UniqueName: \"kubernetes.io/projected/5d70ad2e-86fa-40c7-86e3-98adba4c3913-kube-api-access-bnfc9\") pod \"maas-api-key-cleanup-e2e-worker-w6-14af5c-29889960-ww6xn\" (UID: \"5d70ad2e-86fa-40c7-86e3-98adba4c3913\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-14af5c-29889960-ww6xn"
Oct 30 22:00:00.345493 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:00.345487    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/865d5bf1-9788-4c88-8ae6-e1a20f1efa66-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-5be0d1-29889960-7tvfp\" (UID: \"865d5bf1-9788-4c88-8ae6-e1a20f1efa66\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5be0d1-29889960-7tvfp"
Oct 30 22:00:00.345773 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:00.345604    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d70ad2e-86fa-40c7-86e3-98adba4c3913-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-14af5c-29889960-ww6xn\" (UID: \"5d70ad2e-86fa-40c7-86e3-98adba4c3913\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-14af5c-29889960-ww6xn"
Oct 30 22:00:00.345773 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:00.345672    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2bv4\" (UniqueName: \"kubernetes.io/projected/ccf31bca-ad59-43cc-bf62-effb3add93a0-kube-api-access-x2bv4\") pod \"maas-api-key-cleanup-e2e-worker-w4-17d809-29889960-hwzbr\" (UID: \"ccf31bca-ad59-43cc-bf62-effb3add93a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-17d809-29889960-hwzbr"
Oct 30 22:00:00.345773 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:00.345713    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccf31bca-ad59-43cc-bf62-effb3add93a0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-17d809-29889960-hwzbr\" (UID: \"ccf31bca-ad59-43cc-bf62-effb3add93a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-17d809-29889960-hwzbr"
Oct 30 22:00:00.345773 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:00.345732    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtjxm\" (UniqueName: \"kubernetes.io/projected/865d5bf1-9788-4c88-8ae6-e1a20f1efa66-kube-api-access-vtjxm\") pod \"maas-api-key-cleanup-e2e-worker-w2-5be0d1-29889960-7tvfp\" (UID: \"865d5bf1-9788-4c88-8ae6-e1a20f1efa66\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5be0d1-29889960-7tvfp"
Oct 30 22:00:00.346732 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:00.346710    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/865d5bf1-9788-4c88-8ae6-e1a20f1efa66-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-5be0d1-29889960-7tvfp\" (UID: \"865d5bf1-9788-4c88-8ae6-e1a20f1efa66\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5be0d1-29889960-7tvfp"
Oct 30 22:00:00.347171 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:00.347152    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d70ad2e-86fa-40c7-86e3-98adba4c3913-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-14af5c-29889960-ww6xn\" (UID: \"5d70ad2e-86fa-40c7-86e3-98adba4c3913\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-14af5c-29889960-ww6xn"
Oct 30 22:00:00.401150 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:00.401113    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtjxm\" (UniqueName: \"kubernetes.io/projected/865d5bf1-9788-4c88-8ae6-e1a20f1efa66-kube-api-access-vtjxm\") pod \"maas-api-key-cleanup-e2e-worker-w2-5be0d1-29889960-7tvfp\" (UID: \"865d5bf1-9788-4c88-8ae6-e1a20f1efa66\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5be0d1-29889960-7tvfp"
Oct 30 22:00:00.407149 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:00.407119    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnfc9\" (UniqueName: \"kubernetes.io/projected/5d70ad2e-86fa-40c7-86e3-98adba4c3913-kube-api-access-bnfc9\") pod \"maas-api-key-cleanup-e2e-worker-w6-14af5c-29889960-ww6xn\" (UID: \"5d70ad2e-86fa-40c7-86e3-98adba4c3913\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-14af5c-29889960-ww6xn"
Oct 30 22:00:00.447208 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:00.447114    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccf31bca-ad59-43cc-bf62-effb3add93a0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-17d809-29889960-hwzbr\" (UID: \"ccf31bca-ad59-43cc-bf62-effb3add93a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-17d809-29889960-hwzbr"
Oct 30 22:00:00.447426 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:00.447340    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2bv4\" (UniqueName: \"kubernetes.io/projected/ccf31bca-ad59-43cc-bf62-effb3add93a0-kube-api-access-x2bv4\") pod \"maas-api-key-cleanup-e2e-worker-w4-17d809-29889960-hwzbr\" (UID: \"ccf31bca-ad59-43cc-bf62-effb3add93a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-17d809-29889960-hwzbr"
Oct 30 22:00:00.447791 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:00.447763    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccf31bca-ad59-43cc-bf62-effb3add93a0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-17d809-29889960-hwzbr\" (UID: \"ccf31bca-ad59-43cc-bf62-effb3add93a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-17d809-29889960-hwzbr"
Oct 30 22:00:00.501952 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:00.501922    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2bv4\" (UniqueName: \"kubernetes.io/projected/ccf31bca-ad59-43cc-bf62-effb3add93a0-kube-api-access-x2bv4\") pod \"maas-api-key-cleanup-e2e-worker-w4-17d809-29889960-hwzbr\" (UID: \"ccf31bca-ad59-43cc-bf62-effb3add93a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-17d809-29889960-hwzbr"
Oct 30 22:00:00.507164 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:00.507141    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-14af5c-29889960-ww6xn"
Oct 30 22:00:00.540316 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:00.540288    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5be0d1-29889960-7tvfp"
Oct 30 22:00:00.568719 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:00.568691    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-17d809-29889960-hwzbr"
Oct 30 22:00:00.648736 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:00.637110    5061 patch_prober.go:29] interesting pod/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-9kcgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Oct 30 22:00:00.648736 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:00.637182    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-9kcgf" podUID="9b521eca-5188-43b6-b589-d0ccab8c4520" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Oct 30 22:00:00.936651 ip-10-0-130-60 kubenswrapper[5061]: W1030 22:00:00.936610    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d70ad2e_86fa_40c7_86e3_98adba4c3913.slice/crio-0aeaa5f1d51a7cc679f8d5758035b1c5ec3a3389ef2af3c57a75fcdd5e3742a5 WatchSource:0}: Error finding container 0aeaa5f1d51a7cc679f8d5758035b1c5ec3a3389ef2af3c57a75fcdd5e3742a5: Status 404 returned error can't find the container with id 0aeaa5f1d51a7cc679f8d5758035b1c5ec3a3389ef2af3c57a75fcdd5e3742a5
Oct 30 22:00:00.937938 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:00.937916    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-14af5c-29889960-ww6xn"]
Oct 30 22:00:00.965664 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:00.965637    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5be0d1-29889960-7tvfp"]
Oct 30 22:00:00.969792 ip-10-0-130-60 kubenswrapper[5061]: W1030 22:00:00.969764    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod865d5bf1_9788_4c88_8ae6_e1a20f1efa66.slice/crio-39c2ee201fb721a69e16b0f52a214887514c12e8c75f541ee1f970151b123a23 WatchSource:0}: Error finding container 39c2ee201fb721a69e16b0f52a214887514c12e8c75f541ee1f970151b123a23: Status 404 returned error can't find the container with id 39c2ee201fb721a69e16b0f52a214887514c12e8c75f541ee1f970151b123a23
Oct 30 22:00:01.040718 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:01.040683    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-17d809-29889960-hwzbr"]
Oct 30 22:00:01.099004 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:01.098971    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5be0d1-29889960-7tvfp" event={"ID":"865d5bf1-9788-4c88-8ae6-e1a20f1efa66","Type":"ContainerStarted","Data":"39c2ee201fb721a69e16b0f52a214887514c12e8c75f541ee1f970151b123a23"}
Oct 30 22:00:01.100309 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:01.100283    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-14af5c-29889960-ww6xn" event={"ID":"5d70ad2e-86fa-40c7-86e3-98adba4c3913","Type":"ContainerStarted","Data":"0aeaa5f1d51a7cc679f8d5758035b1c5ec3a3389ef2af3c57a75fcdd5e3742a5"}
Oct 30 22:00:01.118738 ip-10-0-130-60 kubenswrapper[5061]: W1030 22:00:01.118356    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccf31bca_ad59_43cc_bf62_effb3add93a0.slice/crio-a95b1d8d00dcf439109be78cdc023422e47e0851ad5c844624fef91c8f68f156 WatchSource:0}: Error finding container a95b1d8d00dcf439109be78cdc023422e47e0851ad5c844624fef91c8f68f156: Status 404 returned error can't find the container with id a95b1d8d00dcf439109be78cdc023422e47e0851ad5c844624fef91c8f68f156
Oct 30 22:00:01.635765 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:01.635731    5061 patch_prober.go:29] interesting pod/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-9kcgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Oct 30 22:00:01.636142 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:01.635789    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-9kcgf" podUID="9b521eca-5188-43b6-b589-d0ccab8c4520" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Oct 30 22:00:02.112411 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:02.112379    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5be0d1-29889960-7tvfp" event={"ID":"865d5bf1-9788-4c88-8ae6-e1a20f1efa66","Type":"ContainerStarted","Data":"038f40a3009de0dcc34eb3b01fc6b996ebd2fe894c3be7704169bf9595cfceb7"}
Oct 30 22:00:02.114781 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:02.114757    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-17d809-29889960-hwzbr" event={"ID":"ccf31bca-ad59-43cc-bf62-effb3add93a0","Type":"ContainerStarted","Data":"f82476c28d34220a8f635e8b48b11142dbc23b6c3e46c849ffe711f0cdce5e14"}
Oct 30 22:00:02.114909 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:02.114806    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-17d809-29889960-hwzbr" event={"ID":"ccf31bca-ad59-43cc-bf62-effb3add93a0","Type":"ContainerStarted","Data":"a95b1d8d00dcf439109be78cdc023422e47e0851ad5c844624fef91c8f68f156"}
Oct 30 22:00:02.117237 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:02.117212    5061 generic.go:356] "Generic (PLEG): container finished" podID="5d70ad2e-86fa-40c7-86e3-98adba4c3913" containerID="6c5f009dc1c848e49879d71137de89eba64c4bb8f9f4468cb012bba5cc30da86" exitCode=60
Oct 30 22:00:02.117620 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:02.117441    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-14af5c-29889960-ww6xn" event={"ID":"5d70ad2e-86fa-40c7-86e3-98adba4c3913","Type":"ContainerDied","Data":"6c5f009dc1c848e49879d71137de89eba64c4bb8f9f4468cb012bba5cc30da86"}
Oct 30 22:00:02.117793 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:02.117700    5061 scope.go:122] "RemoveContainer" containerID="6c5f009dc1c848e49879d71137de89eba64c4bb8f9f4468cb012bba5cc30da86"
Oct 30 22:00:02.221723 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:02.221662    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5be0d1-29889960-7tvfp" podStartSLOduration=2.22164564 podStartE2EDuration="2.22164564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:02.220475838 +0000 UTC m=+1055.833245599" watchObservedRunningTime="2026-10-30 22:00:02.22164564 +0000 UTC m=+1055.834415400"
Oct 30 22:00:02.334884 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:02.334830    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-17d809-29889960-hwzbr" podStartSLOduration=2.334815032 podStartE2EDuration="2.334815032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:02.331598009 +0000 UTC m=+1055.944367766" watchObservedRunningTime="2026-10-30 22:00:02.334815032 +0000 UTC m=+1055.947584790"
Oct 30 22:00:02.639998 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:02.639955    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-9kcgf"
Oct 30 22:00:02.640487 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:02.640395    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-9kcgf"
Oct 30 22:00:02.641174 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:02.641150    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-9kcgf"
Oct 30 22:00:03.130565 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:03.126686    5061 generic.go:356] "Generic (PLEG): container finished" podID="865d5bf1-9788-4c88-8ae6-e1a20f1efa66" containerID="038f40a3009de0dcc34eb3b01fc6b996ebd2fe894c3be7704169bf9595cfceb7" exitCode=60
Oct 30 22:00:03.130565 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:03.126804    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5be0d1-29889960-7tvfp" event={"ID":"865d5bf1-9788-4c88-8ae6-e1a20f1efa66","Type":"ContainerDied","Data":"038f40a3009de0dcc34eb3b01fc6b996ebd2fe894c3be7704169bf9595cfceb7"}
Oct 30 22:00:03.130565 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:03.127127    5061 scope.go:122] "RemoveContainer" containerID="038f40a3009de0dcc34eb3b01fc6b996ebd2fe894c3be7704169bf9595cfceb7"
Oct 30 22:00:03.130565 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:03.128723    5061 generic.go:356] "Generic (PLEG): container finished" podID="ccf31bca-ad59-43cc-bf62-effb3add93a0" containerID="f82476c28d34220a8f635e8b48b11142dbc23b6c3e46c849ffe711f0cdce5e14" exitCode=60
Oct 30 22:00:03.130565 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:03.128804    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-17d809-29889960-hwzbr" event={"ID":"ccf31bca-ad59-43cc-bf62-effb3add93a0","Type":"ContainerDied","Data":"f82476c28d34220a8f635e8b48b11142dbc23b6c3e46c849ffe711f0cdce5e14"}
Oct 30 22:00:03.130565 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:03.129047    5061 scope.go:122] "RemoveContainer" containerID="f82476c28d34220a8f635e8b48b11142dbc23b6c3e46c849ffe711f0cdce5e14"
Oct 30 22:00:03.131582 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:03.130833    5061 generic.go:356] "Generic (PLEG): container finished" podID="5d70ad2e-86fa-40c7-86e3-98adba4c3913" containerID="c76cb928ef8af01a8811931ecf3b2a12ce17767ba955f762d3284b971af4f70a" exitCode=60
Oct 30 22:00:03.131582 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:03.130944    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-14af5c-29889960-ww6xn" event={"ID":"5d70ad2e-86fa-40c7-86e3-98adba4c3913","Type":"ContainerDied","Data":"c76cb928ef8af01a8811931ecf3b2a12ce17767ba955f762d3284b971af4f70a"}
Oct 30 22:00:03.131582 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:03.131023    5061 scope.go:122] "RemoveContainer" containerID="6c5f009dc1c848e49879d71137de89eba64c4bb8f9f4468cb012bba5cc30da86"
Oct 30 22:00:03.131582 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:03.131138    5061 scope.go:122] "RemoveContainer" containerID="c76cb928ef8af01a8811931ecf3b2a12ce17767ba955f762d3284b971af4f70a"
Oct 30 22:00:03.131954 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:00:03.131930    5061 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-14af5c-29889960-ww6xn_odh-ai-gateway-infra(5d70ad2e-86fa-40c7-86e3-98adba4c3913)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-14af5c-29889960-ww6xn" podUID="5d70ad2e-86fa-40c7-86e3-98adba4c3913"
Oct 30 22:00:03.676425 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:03.676333    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-17d809/distinct-w4-17d809-kserve-6b8f5c48cd-lhrwp"
Oct 30 22:00:03.689426 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:03.689397    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-17d809/distinct-w4-17d809-kserve-6b8f5c48cd-lhrwp"
Oct 30 22:00:04.138724 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:04.138698    5061 scope.go:122] "RemoveContainer" containerID="c76cb928ef8af01a8811931ecf3b2a12ce17767ba955f762d3284b971af4f70a"
Oct 30 22:00:04.139049 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:00:04.139027    5061 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-14af5c-29889960-ww6xn_odh-ai-gateway-infra(5d70ad2e-86fa-40c7-86e3-98adba4c3913)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-14af5c-29889960-ww6xn" podUID="5d70ad2e-86fa-40c7-86e3-98adba4c3913"
Oct 30 22:00:04.140338 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:04.140311    5061 generic.go:356] "Generic (PLEG): container finished" podID="865d5bf1-9788-4c88-8ae6-e1a20f1efa66" containerID="f2a01ace6bc136d444764905d70caa6be5c52c46ef316b7abad38fa90371b59a" exitCode=60
Oct 30 22:00:04.140432 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:04.140398    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5be0d1-29889960-7tvfp" event={"ID":"865d5bf1-9788-4c88-8ae6-e1a20f1efa66","Type":"ContainerDied","Data":"f2a01ace6bc136d444764905d70caa6be5c52c46ef316b7abad38fa90371b59a"}
Oct 30 22:00:04.140494 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:04.140461    5061 scope.go:122] "RemoveContainer" containerID="038f40a3009de0dcc34eb3b01fc6b996ebd2fe894c3be7704169bf9595cfceb7"
Oct 30 22:00:04.140697 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:04.140676    5061 scope.go:122] "RemoveContainer" containerID="f2a01ace6bc136d444764905d70caa6be5c52c46ef316b7abad38fa90371b59a"
Oct 30 22:00:04.140991 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:00:04.140969    5061 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-5be0d1-29889960-7tvfp_odh-ai-gateway-infra(865d5bf1-9788-4c88-8ae6-e1a20f1efa66)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5be0d1-29889960-7tvfp" podUID="865d5bf1-9788-4c88-8ae6-e1a20f1efa66"
Oct 30 22:00:04.142310 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:04.142287    5061 generic.go:356] "Generic (PLEG): container finished" podID="ccf31bca-ad59-43cc-bf62-effb3add93a0" containerID="4ca559212a4e4ca583ebe73f4187589e3dc004233423139bb2e41aa49d56e62d" exitCode=60
Oct 30 22:00:04.142407 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:04.142366    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-17d809-29889960-hwzbr" event={"ID":"ccf31bca-ad59-43cc-bf62-effb3add93a0","Type":"ContainerDied","Data":"4ca559212a4e4ca583ebe73f4187589e3dc004233423139bb2e41aa49d56e62d"}
Oct 30 22:00:04.142625 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:04.142609    5061 scope.go:122] "RemoveContainer" containerID="4ca559212a4e4ca583ebe73f4187589e3dc004233423139bb2e41aa49d56e62d"
Oct 30 22:00:04.142876 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:00:04.142838    5061 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-17d809-29889960-hwzbr_odh-ai-gateway-infra(ccf31bca-ad59-43cc-bf62-effb3add93a0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-17d809-29889960-hwzbr" podUID="ccf31bca-ad59-43cc-bf62-effb3add93a0"
Oct 30 22:00:04.156267 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:04.156247    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-17d809/distinct-w4-17d809-kserve-6b8f5c48cd-lhrwp"
Oct 30 22:00:04.186192 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:04.186134    5061 scope.go:122] "RemoveContainer" containerID="f82476c28d34220a8f635e8b48b11142dbc23b6c3e46c849ffe711f0cdce5e14"
Oct 30 22:00:05.151814 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:05.151787    5061 scope.go:122] "RemoveContainer" containerID="f2a01ace6bc136d444764905d70caa6be5c52c46ef316b7abad38fa90371b59a"
Oct 30 22:00:05.152226 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:00:05.152106    5061 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-5be0d1-29889960-7tvfp_odh-ai-gateway-infra(865d5bf1-9788-4c88-8ae6-e1a20f1efa66)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5be0d1-29889960-7tvfp" podUID="865d5bf1-9788-4c88-8ae6-e1a20f1efa66"
Oct 30 22:00:05.154202 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:05.154160    5061 scope.go:122] "RemoveContainer" containerID="4ca559212a4e4ca583ebe73f4187589e3dc004233423139bb2e41aa49d56e62d"
Oct 30 22:00:05.154484 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:00:05.154461    5061 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-17d809-29889960-hwzbr_odh-ai-gateway-infra(ccf31bca-ad59-43cc-bf62-effb3add93a0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-17d809-29889960-hwzbr" podUID="ccf31bca-ad59-43cc-bf62-effb3add93a0"
Oct 30 22:00:09.111512 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:09.111476    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-mpcvw"]
Oct 30 22:00:09.117135 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:09.117114    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-mpcvw"
Oct 30 22:00:09.220686 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:09.220645    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-mpcvw"]
Oct 30 22:00:09.267546 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:09.267485    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d54aa532-ac30-469a-9d86-5b6e227e78c4-workload-socket\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-mpcvw\" (UID: \"d54aa532-ac30-469a-9d86-5b6e227e78c4\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-mpcvw"
Oct 30 22:00:09.267712 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:09.267566    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7ctg\" (UniqueName: \"kubernetes.io/projected/d54aa532-ac30-469a-9d86-5b6e227e78c4-kube-api-access-h7ctg\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-mpcvw\" (UID: \"d54aa532-ac30-469a-9d86-5b6e227e78c4\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-mpcvw"
Oct 30 22:00:09.267752 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:09.267718    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d54aa532-ac30-469a-9d86-5b6e227e78c4-istiod-ca-cert\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-mpcvw\" (UID: \"d54aa532-ac30-469a-9d86-5b6e227e78c4\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-mpcvw"
Oct 30 22:00:09.267879 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:09.267857    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d54aa532-ac30-469a-9d86-5b6e227e78c4-istio-podinfo\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-mpcvw\" (UID: \"d54aa532-ac30-469a-9d86-5b6e227e78c4\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-mpcvw"
Oct 30 22:00:09.267965 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:09.267949    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d54aa532-ac30-469a-9d86-5b6e227e78c4-istio-envoy\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-mpcvw\" (UID: \"d54aa532-ac30-469a-9d86-5b6e227e78c4\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-mpcvw"
Oct 30 22:00:09.268031 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:09.268016    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d54aa532-ac30-469a-9d86-5b6e227e78c4-istio-data\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-mpcvw\" (UID: \"d54aa532-ac30-469a-9d86-5b6e227e78c4\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-mpcvw"
Oct 30 22:00:09.268111 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:09.268092    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d54aa532-ac30-469a-9d86-5b6e227e78c4-workload-certs\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-mpcvw\" (UID: \"d54aa532-ac30-469a-9d86-5b6e227e78c4\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-mpcvw"
Oct 30 22:00:09.268159 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:09.268142    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d54aa532-ac30-469a-9d86-5b6e227e78c4-credential-socket\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-mpcvw\" (UID: \"d54aa532-ac30-469a-9d86-5b6e227e78c4\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-mpcvw"
Oct 30 22:00:09.268239 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:09.268222    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d54aa532-ac30-469a-9d86-5b6e227e78c4-istio-token\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-mpcvw\" (UID: \"d54aa532-ac30-469a-9d86-5b6e227e78c4\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-mpcvw"
Oct 30 22:00:09.369970 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:09.369882    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d54aa532-ac30-469a-9d86-5b6e227e78c4-workload-certs\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-mpcvw\" (UID: \"d54aa532-ac30-469a-9d86-5b6e227e78c4\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-mpcvw"
Oct 30 22:00:09.369970 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:09.369936    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d54aa532-ac30-469a-9d86-5b6e227e78c4-credential-socket\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-mpcvw\" (UID: \"d54aa532-ac30-469a-9d86-5b6e227e78c4\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-mpcvw"
Oct 30 22:00:09.370177 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:09.369974    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d54aa532-ac30-469a-9d86-5b6e227e78c4-istio-token\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-mpcvw\" (UID: \"d54aa532-ac30-469a-9d86-5b6e227e78c4\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-mpcvw"
Oct 30 22:00:09.370177 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:09.370080    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d54aa532-ac30-469a-9d86-5b6e227e78c4-workload-socket\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-mpcvw\" (UID: \"d54aa532-ac30-469a-9d86-5b6e227e78c4\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-mpcvw"
Oct 30 22:00:09.370177 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:09.370119    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7ctg\" (UniqueName: \"kubernetes.io/projected/d54aa532-ac30-469a-9d86-5b6e227e78c4-kube-api-access-h7ctg\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-mpcvw\" (UID: \"d54aa532-ac30-469a-9d86-5b6e227e78c4\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-mpcvw"
Oct 30 22:00:09.370353 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:09.370186    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d54aa532-ac30-469a-9d86-5b6e227e78c4-istiod-ca-cert\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-mpcvw\" (UID: \"d54aa532-ac30-469a-9d86-5b6e227e78c4\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-mpcvw"
Oct 30 22:00:09.370353 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:09.370279    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d54aa532-ac30-469a-9d86-5b6e227e78c4-istio-podinfo\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-mpcvw\" (UID: \"d54aa532-ac30-469a-9d86-5b6e227e78c4\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-mpcvw"
Oct 30 22:00:09.370353 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:09.370339    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d54aa532-ac30-469a-9d86-5b6e227e78c4-istio-envoy\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-mpcvw\" (UID: \"d54aa532-ac30-469a-9d86-5b6e227e78c4\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-mpcvw"
Oct 30 22:00:09.370516 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:09.370361    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d54aa532-ac30-469a-9d86-5b6e227e78c4-workload-certs\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-mpcvw\" (UID: \"d54aa532-ac30-469a-9d86-5b6e227e78c4\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-mpcvw"
Oct 30 22:00:09.370516 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:09.370388    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d54aa532-ac30-469a-9d86-5b6e227e78c4-istio-data\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-mpcvw\" (UID: \"d54aa532-ac30-469a-9d86-5b6e227e78c4\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-mpcvw"
Oct 30 22:00:09.370516 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:09.370389    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d54aa532-ac30-469a-9d86-5b6e227e78c4-credential-socket\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-mpcvw\" (UID: \"d54aa532-ac30-469a-9d86-5b6e227e78c4\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-mpcvw"
Oct 30 22:00:09.370516 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:09.370483    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d54aa532-ac30-469a-9d86-5b6e227e78c4-workload-socket\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-mpcvw\" (UID: \"d54aa532-ac30-469a-9d86-5b6e227e78c4\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-mpcvw"
Oct 30 22:00:09.370754 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:09.370728    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d54aa532-ac30-469a-9d86-5b6e227e78c4-istio-data\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-mpcvw\" (UID: \"d54aa532-ac30-469a-9d86-5b6e227e78c4\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-mpcvw"
Oct 30 22:00:09.371077 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:09.371058    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d54aa532-ac30-469a-9d86-5b6e227e78c4-istiod-ca-cert\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-mpcvw\" (UID: \"d54aa532-ac30-469a-9d86-5b6e227e78c4\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-mpcvw"
Oct 30 22:00:09.372602 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:09.372579    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d54aa532-ac30-469a-9d86-5b6e227e78c4-istio-envoy\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-mpcvw\" (UID: \"d54aa532-ac30-469a-9d86-5b6e227e78c4\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-mpcvw"
Oct 30 22:00:09.372802 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:09.372784    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d54aa532-ac30-469a-9d86-5b6e227e78c4-istio-podinfo\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-mpcvw\" (UID: \"d54aa532-ac30-469a-9d86-5b6e227e78c4\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-mpcvw"
Oct 30 22:00:09.424357 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:09.424313    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d54aa532-ac30-469a-9d86-5b6e227e78c4-istio-token\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-mpcvw\" (UID: \"d54aa532-ac30-469a-9d86-5b6e227e78c4\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-mpcvw"
Oct 30 22:00:09.424522 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:09.424498    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7ctg\" (UniqueName: \"kubernetes.io/projected/d54aa532-ac30-469a-9d86-5b6e227e78c4-kube-api-access-h7ctg\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-mpcvw\" (UID: \"d54aa532-ac30-469a-9d86-5b6e227e78c4\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-mpcvw"
Oct 30 22:00:09.433501 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:09.433436    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-mpcvw"
Oct 30 22:00:09.865371 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:09.865337    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-mpcvw"]
Oct 30 22:00:09.868838 ip-10-0-130-60 kubenswrapper[5061]: W1030 22:00:09.868807    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd54aa532_ac30_469a_9d86_5b6e227e78c4.slice/crio-8074d848b729fbccf2404b685568e452a984639650ad7549d94dd7e063dd1e43 WatchSource:0}: Error finding container 8074d848b729fbccf2404b685568e452a984639650ad7549d94dd7e063dd1e43: Status 404 returned error can't find the container with id 8074d848b729fbccf2404b685568e452a984639650ad7549d94dd7e063dd1e43
Oct 30 22:00:09.871461 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:09.871420    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:00:09.871564 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:09.871493    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:00:09.871564 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:09.871522    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:00:10.213469 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:10.213374    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-mpcvw" event={"ID":"d54aa532-ac30-469a-9d86-5b6e227e78c4","Type":"ContainerStarted","Data":"077c79aa167cffafb2f96a63f3e8416d1d139c8083321366b77da2fdf9b4f998"}
Oct 30 22:00:10.214103 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:10.214077    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-mpcvw" event={"ID":"d54aa532-ac30-469a-9d86-5b6e227e78c4","Type":"ContainerStarted","Data":"8074d848b729fbccf2404b685568e452a984639650ad7549d94dd7e063dd1e43"}
Oct 30 22:00:10.329485 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:10.329412    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-mpcvw" podStartSLOduration=1.329391483 podStartE2EDuration="1.329391483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:10.325254811 +0000 UTC m=+1063.938024589" watchObservedRunningTime="2026-10-30 22:00:10.329391483 +0000 UTC m=+1063.942161244"
Oct 30 22:00:10.433811 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:10.433775    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-mpcvw"
Oct 30 22:00:10.435465 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:10.435435    5061 patch_prober.go:29] interesting pod/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-mpcvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Oct 30 22:00:10.435608 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:10.435489    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-mpcvw" podUID="d54aa532-ac30-469a-9d86-5b6e227e78c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Oct 30 22:00:11.434090 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:11.434049    5061 patch_prober.go:29] interesting pod/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-mpcvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Oct 30 22:00:11.434541 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:11.434116    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-mpcvw" podUID="d54aa532-ac30-469a-9d86-5b6e227e78c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Oct 30 22:00:12.434032 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:12.433996    5061 patch_prober.go:29] interesting pod/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-mpcvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Oct 30 22:00:12.434189 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:12.434060    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-mpcvw" podUID="d54aa532-ac30-469a-9d86-5b6e227e78c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Oct 30 22:00:13.434198 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:13.434156    5061 patch_prober.go:29] interesting pod/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-mpcvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Oct 30 22:00:13.434721 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:13.434247    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-mpcvw" podUID="d54aa532-ac30-469a-9d86-5b6e227e78c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Oct 30 22:00:14.434032 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:14.433998    5061 patch_prober.go:29] interesting pod/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-mpcvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Oct 30 22:00:14.434207 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:14.434050    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-mpcvw" podUID="d54aa532-ac30-469a-9d86-5b6e227e78c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Oct 30 22:00:15.434047 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:15.434006    5061 patch_prober.go:29] interesting pod/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-mpcvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Oct 30 22:00:15.434471 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:15.434079    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-mpcvw" podUID="d54aa532-ac30-469a-9d86-5b6e227e78c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Oct 30 22:00:15.524564 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:15.524139    5061 scope.go:122] "RemoveContainer" containerID="c76cb928ef8af01a8811931ecf3b2a12ce17767ba955f762d3284b971af4f70a"
Oct 30 22:00:16.276922 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:16.276887    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-14af5c-29889960-ww6xn" event={"ID":"5d70ad2e-86fa-40c7-86e3-98adba4c3913","Type":"ContainerStarted","Data":"0325e9767d16c30076ed56328376403052ae2f2d0b5e7866cdfed8903e293d17"}
Oct 30 22:00:16.387064 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:16.386985    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-14af5c-29889960-ww6xn" podStartSLOduration=16.386964729 podStartE2EDuration="16.386964729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:16.385546813 +0000 UTC m=+1069.998316569" watchObservedRunningTime="2026-10-30 22:00:16.386964729 +0000 UTC m=+1069.999734487"
Oct 30 22:00:16.434601 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:16.434568    5061 patch_prober.go:29] interesting pod/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-mpcvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Oct 30 22:00:16.435012 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:16.434671    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-mpcvw" podUID="d54aa532-ac30-469a-9d86-5b6e227e78c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Oct 30 22:00:16.525731 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:16.525697    5061 scope.go:122] "RemoveContainer" containerID="f2a01ace6bc136d444764905d70caa6be5c52c46ef316b7abad38fa90371b59a"
Oct 30 22:00:17.285976 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:17.285889    5061 generic.go:356] "Generic (PLEG): container finished" podID="5d70ad2e-86fa-40c7-86e3-98adba4c3913" containerID="0325e9767d16c30076ed56328376403052ae2f2d0b5e7866cdfed8903e293d17" exitCode=60
Oct 30 22:00:17.286151 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:17.285965    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-14af5c-29889960-ww6xn" event={"ID":"5d70ad2e-86fa-40c7-86e3-98adba4c3913","Type":"ContainerDied","Data":"0325e9767d16c30076ed56328376403052ae2f2d0b5e7866cdfed8903e293d17"}
Oct 30 22:00:17.286151 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:17.286038    5061 scope.go:122] "RemoveContainer" containerID="c76cb928ef8af01a8811931ecf3b2a12ce17767ba955f762d3284b971af4f70a"
Oct 30 22:00:17.287021 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:17.286997    5061 scope.go:122] "RemoveContainer" containerID="0325e9767d16c30076ed56328376403052ae2f2d0b5e7866cdfed8903e293d17"
Oct 30 22:00:17.287364 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:00:17.287341    5061 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-14af5c-29889960-ww6xn_odh-ai-gateway-infra(5d70ad2e-86fa-40c7-86e3-98adba4c3913)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-14af5c-29889960-ww6xn" podUID="5d70ad2e-86fa-40c7-86e3-98adba4c3913"
Oct 30 22:00:17.288301 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:17.288283    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5be0d1-29889960-7tvfp" event={"ID":"865d5bf1-9788-4c88-8ae6-e1a20f1efa66","Type":"ContainerStarted","Data":"0696d134480d45e3c205f591777c11d7698014b436df911582992074ba3a4256"}
Oct 30 22:00:17.398775 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:17.398740    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-14af5c-29889960-ww6xn"]
Oct 30 22:00:17.438820 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:17.438792    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-mpcvw"
Oct 30 22:00:17.439202 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:17.439077    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-mpcvw"
Oct 30 22:00:17.439840 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:17.439821    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-mpcvw"
Oct 30 22:00:17.522026 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:17.521988    5061 scope.go:122] "RemoveContainer" containerID="4ca559212a4e4ca583ebe73f4187589e3dc004233423139bb2e41aa49d56e62d"
Oct 30 22:00:18.297261 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:18.297230    5061 generic.go:356] "Generic (PLEG): container finished" podID="865d5bf1-9788-4c88-8ae6-e1a20f1efa66" containerID="0696d134480d45e3c205f591777c11d7698014b436df911582992074ba3a4256" exitCode=60
Oct 30 22:00:18.297441 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:18.297297    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5be0d1-29889960-7tvfp" event={"ID":"865d5bf1-9788-4c88-8ae6-e1a20f1efa66","Type":"ContainerDied","Data":"0696d134480d45e3c205f591777c11d7698014b436df911582992074ba3a4256"}
Oct 30 22:00:18.297441 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:18.297394    5061 scope.go:122] "RemoveContainer" containerID="f2a01ace6bc136d444764905d70caa6be5c52c46ef316b7abad38fa90371b59a"
Oct 30 22:00:18.297921 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:18.297899    5061 scope.go:122] "RemoveContainer" containerID="0696d134480d45e3c205f591777c11d7698014b436df911582992074ba3a4256"
Oct 30 22:00:18.298293 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:00:18.298270    5061 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-5be0d1-29889960-7tvfp_odh-ai-gateway-infra(865d5bf1-9788-4c88-8ae6-e1a20f1efa66)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5be0d1-29889960-7tvfp" podUID="865d5bf1-9788-4c88-8ae6-e1a20f1efa66"
Oct 30 22:00:18.299521 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:18.299494    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-17d809-29889960-hwzbr" event={"ID":"ccf31bca-ad59-43cc-bf62-effb3add93a0","Type":"ContainerStarted","Data":"cac3f22a011a4b7cf096954808543c0f3f9e7aad567bb377162ec368ccd851ee"}
Oct 30 22:00:18.459108 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:18.459083    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-14af5c-29889960-ww6xn"
Oct 30 22:00:18.519337 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:18.519313    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d70ad2e-86fa-40c7-86e3-98adba4c3913-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d70ad2e-86fa-40c7-86e3-98adba4c3913-maas-api-service-ca\") pod \"5d70ad2e-86fa-40c7-86e3-98adba4c3913\" (UID: \"5d70ad2e-86fa-40c7-86e3-98adba4c3913\") "
Oct 30 22:00:18.519517 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:18.519504    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d70ad2e-86fa-40c7-86e3-98adba4c3913-kube-api-access-bnfc9\" (UniqueName: \"kubernetes.io/projected/5d70ad2e-86fa-40c7-86e3-98adba4c3913-kube-api-access-bnfc9\") pod \"5d70ad2e-86fa-40c7-86e3-98adba4c3913\" (UID: \"5d70ad2e-86fa-40c7-86e3-98adba4c3913\") "
Oct 30 22:00:18.519745 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:18.519716    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d70ad2e-86fa-40c7-86e3-98adba4c3913-maas-api-service-ca" pod "5d70ad2e-86fa-40c7-86e3-98adba4c3913" (UID: "5d70ad2e-86fa-40c7-86e3-98adba4c3913"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:00:18.520041 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:18.520023    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d70ad2e-86fa-40c7-86e3-98adba4c3913-maas-api-service-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:00:18.521816 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:18.521792    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d70ad2e-86fa-40c7-86e3-98adba4c3913-kube-api-access-bnfc9" pod "5d70ad2e-86fa-40c7-86e3-98adba4c3913" (UID: "5d70ad2e-86fa-40c7-86e3-98adba4c3913"). InnerVolumeSpecName "kube-api-access-bnfc9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:00:18.529780 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:18.529757    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5be0d1-29889960-7tvfp"]
Oct 30 22:00:18.621468 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:18.621426    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnfc9\" (UniqueName: \"kubernetes.io/projected/5d70ad2e-86fa-40c7-86e3-98adba4c3913-kube-api-access-bnfc9\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:00:19.309441 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:19.309404    5061 generic.go:356] "Generic (PLEG): container finished" podID="ccf31bca-ad59-43cc-bf62-effb3add93a0" containerID="cac3f22a011a4b7cf096954808543c0f3f9e7aad567bb377162ec368ccd851ee" exitCode=60
Oct 30 22:00:19.309638 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:19.309473    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-17d809-29889960-hwzbr" event={"ID":"ccf31bca-ad59-43cc-bf62-effb3add93a0","Type":"ContainerDied","Data":"cac3f22a011a4b7cf096954808543c0f3f9e7aad567bb377162ec368ccd851ee"}
Oct 30 22:00:19.309638 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:19.309544    5061 scope.go:122] "RemoveContainer" containerID="4ca559212a4e4ca583ebe73f4187589e3dc004233423139bb2e41aa49d56e62d"
Oct 30 22:00:19.309958 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:19.309938    5061 scope.go:122] "RemoveContainer" containerID="cac3f22a011a4b7cf096954808543c0f3f9e7aad567bb377162ec368ccd851ee"
Oct 30 22:00:19.310324 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:00:19.310295    5061 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-17d809-29889960-hwzbr_odh-ai-gateway-infra(ccf31bca-ad59-43cc-bf62-effb3add93a0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-17d809-29889960-hwzbr" podUID="ccf31bca-ad59-43cc-bf62-effb3add93a0"
Oct 30 22:00:19.311119 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:19.311102    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-14af5c-29889960-ww6xn"
Oct 30 22:00:19.311197 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:19.311132    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-14af5c-29889960-ww6xn" event={"ID":"5d70ad2e-86fa-40c7-86e3-98adba4c3913","Type":"ContainerDied","Data":"0aeaa5f1d51a7cc679f8d5758035b1c5ec3a3389ef2af3c57a75fcdd5e3742a5"}
Oct 30 22:00:19.322841 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:19.322820    5061 scope.go:122] "RemoveContainer" containerID="0325e9767d16c30076ed56328376403052ae2f2d0b5e7866cdfed8903e293d17"
Oct 30 22:00:19.469183 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:19.469156    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5be0d1-29889960-7tvfp"
Oct 30 22:00:19.527587 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:19.527550    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-14af5c-29889960-ww6xn"]
Oct 30 22:00:19.530882 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:19.530852    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-14af5c-29889960-ww6xn"]
Oct 30 22:00:19.534418 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:19.534392    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/865d5bf1-9788-4c88-8ae6-e1a20f1efa66-kube-api-access-vtjxm\" (UniqueName: \"kubernetes.io/projected/865d5bf1-9788-4c88-8ae6-e1a20f1efa66-kube-api-access-vtjxm\") pod \"865d5bf1-9788-4c88-8ae6-e1a20f1efa66\" (UID: \"865d5bf1-9788-4c88-8ae6-e1a20f1efa66\") "
Oct 30 22:00:19.534561 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:19.534457    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/865d5bf1-9788-4c88-8ae6-e1a20f1efa66-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/865d5bf1-9788-4c88-8ae6-e1a20f1efa66-maas-api-service-ca\") pod \"865d5bf1-9788-4c88-8ae6-e1a20f1efa66\" (UID: \"865d5bf1-9788-4c88-8ae6-e1a20f1efa66\") "
Oct 30 22:00:19.534866 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:19.534843    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/865d5bf1-9788-4c88-8ae6-e1a20f1efa66-maas-api-service-ca" pod "865d5bf1-9788-4c88-8ae6-e1a20f1efa66" (UID: "865d5bf1-9788-4c88-8ae6-e1a20f1efa66"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:00:19.535014 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:19.534998    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/865d5bf1-9788-4c88-8ae6-e1a20f1efa66-maas-api-service-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:00:19.536632 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:19.536607    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/865d5bf1-9788-4c88-8ae6-e1a20f1efa66-kube-api-access-vtjxm" pod "865d5bf1-9788-4c88-8ae6-e1a20f1efa66" (UID: "865d5bf1-9788-4c88-8ae6-e1a20f1efa66"). InnerVolumeSpecName "kube-api-access-vtjxm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:00:19.636958 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:19.636878    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtjxm\" (UniqueName: \"kubernetes.io/projected/865d5bf1-9788-4c88-8ae6-e1a20f1efa66-kube-api-access-vtjxm\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:00:19.642365 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:19.642335    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-17d809-29889960-hwzbr"]
Oct 30 22:00:20.322467 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:20.322435    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5be0d1-29889960-7tvfp"
Oct 30 22:00:20.322467 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:20.322444    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5be0d1-29889960-7tvfp" event={"ID":"865d5bf1-9788-4c88-8ae6-e1a20f1efa66","Type":"ContainerDied","Data":"39c2ee201fb721a69e16b0f52a214887514c12e8c75f541ee1f970151b123a23"}
Oct 30 22:00:20.322755 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:20.322500    5061 scope.go:122] "RemoveContainer" containerID="0696d134480d45e3c205f591777c11d7698014b436df911582992074ba3a4256"
Oct 30 22:00:20.483944 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:20.483752    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5be0d1-29889960-7tvfp"]
Oct 30 22:00:20.489079 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:20.489051    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5be0d1-29889960-7tvfp"]
Oct 30 22:00:20.506843 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:20.506819    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-17d809-29889960-hwzbr"
Oct 30 22:00:20.530958 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:20.530925    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d70ad2e-86fa-40c7-86e3-98adba4c3913" path="/var/lib/kubelet/pods/5d70ad2e-86fa-40c7-86e3-98adba4c3913/volumes"
Oct 30 22:00:20.531462 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:20.531442    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="865d5bf1-9788-4c88-8ae6-e1a20f1efa66" path="/var/lib/kubelet/pods/865d5bf1-9788-4c88-8ae6-e1a20f1efa66/volumes"
Oct 30 22:00:20.547686 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:20.547660    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ccf31bca-ad59-43cc-bf62-effb3add93a0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccf31bca-ad59-43cc-bf62-effb3add93a0-maas-api-service-ca\") pod \"ccf31bca-ad59-43cc-bf62-effb3add93a0\" (UID: \"ccf31bca-ad59-43cc-bf62-effb3add93a0\") "
Oct 30 22:00:20.547868 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:20.547856    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccf31bca-ad59-43cc-bf62-effb3add93a0-kube-api-access-x2bv4\" (UniqueName: \"kubernetes.io/projected/ccf31bca-ad59-43cc-bf62-effb3add93a0-kube-api-access-x2bv4\") pod \"ccf31bca-ad59-43cc-bf62-effb3add93a0\" (UID: \"ccf31bca-ad59-43cc-bf62-effb3add93a0\") "
Oct 30 22:00:20.547999 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:20.547978    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccf31bca-ad59-43cc-bf62-effb3add93a0-maas-api-service-ca" pod "ccf31bca-ad59-43cc-bf62-effb3add93a0" (UID: "ccf31bca-ad59-43cc-bf62-effb3add93a0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:00:20.548232 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:20.548219    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccf31bca-ad59-43cc-bf62-effb3add93a0-maas-api-service-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:00:20.549895 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:20.549879    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccf31bca-ad59-43cc-bf62-effb3add93a0-kube-api-access-x2bv4" pod "ccf31bca-ad59-43cc-bf62-effb3add93a0" (UID: "ccf31bca-ad59-43cc-bf62-effb3add93a0"). InnerVolumeSpecName "kube-api-access-x2bv4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:00:20.650577 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:20.650455    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2bv4\" (UniqueName: \"kubernetes.io/projected/ccf31bca-ad59-43cc-bf62-effb3add93a0-kube-api-access-x2bv4\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:00:21.333687 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:21.333653    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-17d809-29889960-hwzbr"
Oct 30 22:00:21.333872 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:21.333699    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-17d809-29889960-hwzbr" event={"ID":"ccf31bca-ad59-43cc-bf62-effb3add93a0","Type":"ContainerDied","Data":"a95b1d8d00dcf439109be78cdc023422e47e0851ad5c844624fef91c8f68f156"}
Oct 30 22:00:21.333872 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:21.333793    5061 scope.go:122] "RemoveContainer" containerID="cac3f22a011a4b7cf096954808543c0f3f9e7aad567bb377162ec368ccd851ee"
Oct 30 22:00:21.491064 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:21.491025    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-17d809-29889960-hwzbr"]
Oct 30 22:00:21.494891 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:21.494862    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-17d809-29889960-hwzbr"]
Oct 30 22:00:22.529662 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:22.529631    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccf31bca-ad59-43cc-bf62-effb3add93a0" path="/var/lib/kubelet/pods/ccf31bca-ad59-43cc-bf62-effb3add93a0/volumes"
Oct 30 22:00:24.668255 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:00:24.668176    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 22:00:41.367245 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:41.367202    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-14af5c/distinct-2-w6-14af5c-kserve-58d554bdb6-r5r6j"]
Oct 30 22:00:41.367904 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:41.367887    5061 cpu_manager.go:414] "containerMap: removing container" podUID="865d5bf1-9788-4c88-8ae6-e1a20f1efa66" containerName="cleanup"
Oct 30 22:00:41.367904 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:41.367905    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="865d5bf1-9788-4c88-8ae6-e1a20f1efa66" containerName="cleanup"
Oct 30 22:00:41.367986 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:41.367914    5061 cpu_manager.go:414] "containerMap: removing container" podUID="ccf31bca-ad59-43cc-bf62-effb3add93a0" containerName="cleanup"
Oct 30 22:00:41.367986 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:41.367919    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccf31bca-ad59-43cc-bf62-effb3add93a0" containerName="cleanup"
Oct 30 22:00:41.367986 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:41.367929    5061 cpu_manager.go:414] "containerMap: removing container" podUID="5d70ad2e-86fa-40c7-86e3-98adba4c3913" containerName="cleanup"
Oct 30 22:00:41.367986 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:41.367935    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d70ad2e-86fa-40c7-86e3-98adba4c3913" containerName="cleanup"
Oct 30 22:00:41.367986 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:41.367953    5061 cpu_manager.go:414] "containerMap: removing container" podUID="865d5bf1-9788-4c88-8ae6-e1a20f1efa66" containerName="cleanup"
Oct 30 22:00:41.367986 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:41.367958    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="865d5bf1-9788-4c88-8ae6-e1a20f1efa66" containerName="cleanup"
Oct 30 22:00:41.367986 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:41.367965    5061 cpu_manager.go:414] "containerMap: removing container" podUID="5d70ad2e-86fa-40c7-86e3-98adba4c3913" containerName="cleanup"
Oct 30 22:00:41.367986 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:41.367970    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d70ad2e-86fa-40c7-86e3-98adba4c3913" containerName="cleanup"
Oct 30 22:00:41.367986 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:41.367979    5061 cpu_manager.go:414] "containerMap: removing container" podUID="865d5bf1-9788-4c88-8ae6-e1a20f1efa66" containerName="cleanup"
Oct 30 22:00:41.367986 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:41.367983    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="865d5bf1-9788-4c88-8ae6-e1a20f1efa66" containerName="cleanup"
Oct 30 22:00:41.368440 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:41.368001    5061 cpu_manager.go:414] "containerMap: removing container" podUID="ccf31bca-ad59-43cc-bf62-effb3add93a0" containerName="cleanup"
Oct 30 22:00:41.368440 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:41.368006    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccf31bca-ad59-43cc-bf62-effb3add93a0" containerName="cleanup"
Oct 30 22:00:41.368440 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:41.368020    5061 cpu_manager.go:414] "containerMap: removing container" podUID="5d70ad2e-86fa-40c7-86e3-98adba4c3913" containerName="cleanup"
Oct 30 22:00:41.368440 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:41.368025    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d70ad2e-86fa-40c7-86e3-98adba4c3913" containerName="cleanup"
Oct 30 22:00:41.368440 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:41.368116    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="865d5bf1-9788-4c88-8ae6-e1a20f1efa66" containerName="cleanup"
Oct 30 22:00:41.368440 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:41.368124    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d70ad2e-86fa-40c7-86e3-98adba4c3913" containerName="cleanup"
Oct 30 22:00:41.368440 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:41.368130    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="865d5bf1-9788-4c88-8ae6-e1a20f1efa66" containerName="cleanup"
Oct 30 22:00:41.368440 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:41.368139    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccf31bca-ad59-43cc-bf62-effb3add93a0" containerName="cleanup"
Oct 30 22:00:41.368440 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:41.368144    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccf31bca-ad59-43cc-bf62-effb3add93a0" containerName="cleanup"
Oct 30 22:00:41.368440 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:41.368151    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccf31bca-ad59-43cc-bf62-effb3add93a0" containerName="cleanup"
Oct 30 22:00:41.368440 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:41.368157    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d70ad2e-86fa-40c7-86e3-98adba4c3913" containerName="cleanup"
Oct 30 22:00:41.373456 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:41.373435    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-14af5c/distinct-2-w6-14af5c-kserve-58d554bdb6-r5r6j"
Oct 30 22:00:41.423645 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:41.423614    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-14af5c\"/\"distinct-2-w6-14af5c-kserve-self-signed-certs\""
Oct 30 22:00:41.455709 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:41.455662    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/990f4e94-708e-45f3-b2ad-01cd50b3257b-model-cache\") pod \"distinct-2-w6-14af5c-kserve-58d554bdb6-r5r6j\" (UID: \"990f4e94-708e-45f3-b2ad-01cd50b3257b\") " pod="e2e-models-e2e-worker-w6-14af5c/distinct-2-w6-14af5c-kserve-58d554bdb6-r5r6j"
Oct 30 22:00:41.455905 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:41.455716    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqqsw\" (UniqueName: \"kubernetes.io/projected/990f4e94-708e-45f3-b2ad-01cd50b3257b-kube-api-access-hqqsw\") pod \"distinct-2-w6-14af5c-kserve-58d554bdb6-r5r6j\" (UID: \"990f4e94-708e-45f3-b2ad-01cd50b3257b\") " pod="e2e-models-e2e-worker-w6-14af5c/distinct-2-w6-14af5c-kserve-58d554bdb6-r5r6j"
Oct 30 22:00:41.455964 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:41.455942    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/990f4e94-708e-45f3-b2ad-01cd50b3257b-tmp-dir\") pod \"distinct-2-w6-14af5c-kserve-58d554bdb6-r5r6j\" (UID: \"990f4e94-708e-45f3-b2ad-01cd50b3257b\") " pod="e2e-models-e2e-worker-w6-14af5c/distinct-2-w6-14af5c-kserve-58d554bdb6-r5r6j"
Oct 30 22:00:41.456101 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:41.456074    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/990f4e94-708e-45f3-b2ad-01cd50b3257b-home\") pod \"distinct-2-w6-14af5c-kserve-58d554bdb6-r5r6j\" (UID: \"990f4e94-708e-45f3-b2ad-01cd50b3257b\") " pod="e2e-models-e2e-worker-w6-14af5c/distinct-2-w6-14af5c-kserve-58d554bdb6-r5r6j"
Oct 30 22:00:41.456257 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:41.456161    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/990f4e94-708e-45f3-b2ad-01cd50b3257b-dshm\") pod \"distinct-2-w6-14af5c-kserve-58d554bdb6-r5r6j\" (UID: \"990f4e94-708e-45f3-b2ad-01cd50b3257b\") " pod="e2e-models-e2e-worker-w6-14af5c/distinct-2-w6-14af5c-kserve-58d554bdb6-r5r6j"
Oct 30 22:00:41.456257 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:41.456190    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/990f4e94-708e-45f3-b2ad-01cd50b3257b-tls-certs\") pod \"distinct-2-w6-14af5c-kserve-58d554bdb6-r5r6j\" (UID: \"990f4e94-708e-45f3-b2ad-01cd50b3257b\") " pod="e2e-models-e2e-worker-w6-14af5c/distinct-2-w6-14af5c-kserve-58d554bdb6-r5r6j"
Oct 30 22:00:41.474678 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:41.474647    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-14af5c/distinct-2-w6-14af5c-kserve-58d554bdb6-r5r6j"]
Oct 30 22:00:41.557468 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:41.557423    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/990f4e94-708e-45f3-b2ad-01cd50b3257b-tmp-dir\") pod \"distinct-2-w6-14af5c-kserve-58d554bdb6-r5r6j\" (UID: \"990f4e94-708e-45f3-b2ad-01cd50b3257b\") " pod="e2e-models-e2e-worker-w6-14af5c/distinct-2-w6-14af5c-kserve-58d554bdb6-r5r6j"
Oct 30 22:00:41.557690 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:41.557513    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/990f4e94-708e-45f3-b2ad-01cd50b3257b-home\") pod \"distinct-2-w6-14af5c-kserve-58d554bdb6-r5r6j\" (UID: \"990f4e94-708e-45f3-b2ad-01cd50b3257b\") " pod="e2e-models-e2e-worker-w6-14af5c/distinct-2-w6-14af5c-kserve-58d554bdb6-r5r6j"
Oct 30 22:00:41.557690 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:41.557577    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/990f4e94-708e-45f3-b2ad-01cd50b3257b-dshm\") pod \"distinct-2-w6-14af5c-kserve-58d554bdb6-r5r6j\" (UID: \"990f4e94-708e-45f3-b2ad-01cd50b3257b\") " pod="e2e-models-e2e-worker-w6-14af5c/distinct-2-w6-14af5c-kserve-58d554bdb6-r5r6j"
Oct 30 22:00:41.557690 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:41.557595    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/990f4e94-708e-45f3-b2ad-01cd50b3257b-tls-certs\") pod \"distinct-2-w6-14af5c-kserve-58d554bdb6-r5r6j\" (UID: \"990f4e94-708e-45f3-b2ad-01cd50b3257b\") " pod="e2e-models-e2e-worker-w6-14af5c/distinct-2-w6-14af5c-kserve-58d554bdb6-r5r6j"
Oct 30 22:00:41.557863 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:41.557716    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/990f4e94-708e-45f3-b2ad-01cd50b3257b-model-cache\") pod \"distinct-2-w6-14af5c-kserve-58d554bdb6-r5r6j\" (UID: \"990f4e94-708e-45f3-b2ad-01cd50b3257b\") " pod="e2e-models-e2e-worker-w6-14af5c/distinct-2-w6-14af5c-kserve-58d554bdb6-r5r6j"
Oct 30 22:00:41.557863 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:41.557741    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqqsw\" (UniqueName: \"kubernetes.io/projected/990f4e94-708e-45f3-b2ad-01cd50b3257b-kube-api-access-hqqsw\") pod \"distinct-2-w6-14af5c-kserve-58d554bdb6-r5r6j\" (UID: \"990f4e94-708e-45f3-b2ad-01cd50b3257b\") " pod="e2e-models-e2e-worker-w6-14af5c/distinct-2-w6-14af5c-kserve-58d554bdb6-r5r6j"
Oct 30 22:00:41.557998 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:41.557966    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/990f4e94-708e-45f3-b2ad-01cd50b3257b-home\") pod \"distinct-2-w6-14af5c-kserve-58d554bdb6-r5r6j\" (UID: \"990f4e94-708e-45f3-b2ad-01cd50b3257b\") " pod="e2e-models-e2e-worker-w6-14af5c/distinct-2-w6-14af5c-kserve-58d554bdb6-r5r6j"
Oct 30 22:00:41.558061 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:41.557976    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/990f4e94-708e-45f3-b2ad-01cd50b3257b-tmp-dir\") pod \"distinct-2-w6-14af5c-kserve-58d554bdb6-r5r6j\" (UID: \"990f4e94-708e-45f3-b2ad-01cd50b3257b\") " pod="e2e-models-e2e-worker-w6-14af5c/distinct-2-w6-14af5c-kserve-58d554bdb6-r5r6j"
Oct 30 22:00:41.558134 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:41.558073    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/990f4e94-708e-45f3-b2ad-01cd50b3257b-model-cache\") pod \"distinct-2-w6-14af5c-kserve-58d554bdb6-r5r6j\" (UID: \"990f4e94-708e-45f3-b2ad-01cd50b3257b\") " pod="e2e-models-e2e-worker-w6-14af5c/distinct-2-w6-14af5c-kserve-58d554bdb6-r5r6j"
Oct 30 22:00:41.559935 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:41.559911    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/990f4e94-708e-45f3-b2ad-01cd50b3257b-dshm\") pod \"distinct-2-w6-14af5c-kserve-58d554bdb6-r5r6j\" (UID: \"990f4e94-708e-45f3-b2ad-01cd50b3257b\") " pod="e2e-models-e2e-worker-w6-14af5c/distinct-2-w6-14af5c-kserve-58d554bdb6-r5r6j"
Oct 30 22:00:41.560318 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:41.560298    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/990f4e94-708e-45f3-b2ad-01cd50b3257b-tls-certs\") pod \"distinct-2-w6-14af5c-kserve-58d554bdb6-r5r6j\" (UID: \"990f4e94-708e-45f3-b2ad-01cd50b3257b\") " pod="e2e-models-e2e-worker-w6-14af5c/distinct-2-w6-14af5c-kserve-58d554bdb6-r5r6j"
Oct 30 22:00:41.612659 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:41.612618    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqqsw\" (UniqueName: \"kubernetes.io/projected/990f4e94-708e-45f3-b2ad-01cd50b3257b-kube-api-access-hqqsw\") pod \"distinct-2-w6-14af5c-kserve-58d554bdb6-r5r6j\" (UID: \"990f4e94-708e-45f3-b2ad-01cd50b3257b\") " pod="e2e-models-e2e-worker-w6-14af5c/distinct-2-w6-14af5c-kserve-58d554bdb6-r5r6j"
Oct 30 22:00:41.685791 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:41.685694    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-14af5c/distinct-2-w6-14af5c-kserve-58d554bdb6-r5r6j"
Oct 30 22:00:42.097487 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:42.097461    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-14af5c/distinct-2-w6-14af5c-kserve-58d554bdb6-r5r6j"]
Oct 30 22:00:42.100220 ip-10-0-130-60 kubenswrapper[5061]: W1030 22:00:42.100189    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod990f4e94_708e_45f3_b2ad_01cd50b3257b.slice/crio-b8ff36995f8badaa5e5c043ec4da00e517b557d03abdccc5954f223e69b370d8 WatchSource:0}: Error finding container b8ff36995f8badaa5e5c043ec4da00e517b557d03abdccc5954f223e69b370d8: Status 404 returned error can't find the container with id b8ff36995f8badaa5e5c043ec4da00e517b557d03abdccc5954f223e69b370d8
Oct 30 22:00:42.472701 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:42.472599    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-14af5c/distinct-2-w6-14af5c-kserve-58d554bdb6-r5r6j" event={"ID":"990f4e94-708e-45f3-b2ad-01cd50b3257b","Type":"ContainerStarted","Data":"5e7910050fbfcac88f9f0834d65fc867bbb60d23395ae586e81639e7953f80cd"}
Oct 30 22:00:42.472701 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:42.472656    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-14af5c/distinct-2-w6-14af5c-kserve-58d554bdb6-r5r6j" event={"ID":"990f4e94-708e-45f3-b2ad-01cd50b3257b","Type":"ContainerStarted","Data":"b8ff36995f8badaa5e5c043ec4da00e517b557d03abdccc5954f223e69b370d8"}
Oct 30 22:00:42.582295 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:42.582229    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-14af5c/distinct-2-w6-14af5c-kserve-58d554bdb6-r5r6j" podStartSLOduration=1.582201844 podStartE2EDuration="1.582201844s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:42.581232364 +0000 UTC m=+1096.194002122" watchObservedRunningTime="2026-10-30 22:00:42.582201844 +0000 UTC m=+1096.194971604"
Oct 30 22:00:42.686330 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:42.686288    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-14af5c/distinct-2-w6-14af5c-kserve-58d554bdb6-r5r6j"
Oct 30 22:00:51.686770 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:51.686736    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-14af5c/distinct-2-w6-14af5c-kserve-58d554bdb6-r5r6j"
Oct 30 22:00:51.699155 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:51.699130    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-14af5c/distinct-2-w6-14af5c-kserve-58d554bdb6-r5r6j"
Oct 30 22:00:52.550928 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:00:52.550897    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-14af5c/distinct-2-w6-14af5c-kserve-58d554bdb6-r5r6j"
Oct 30 22:00:54.669012 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:00:54.668972    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 22:01:23.244924 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:01:23.244883    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-f95pk"]
Oct 30 22:01:23.246100 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:01:23.246073    5061 cpu_manager.go:414] "containerMap: removing container" podUID="ccf31bca-ad59-43cc-bf62-effb3add93a0" containerName="cleanup"
Oct 30 22:01:23.246100 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:01:23.246100    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccf31bca-ad59-43cc-bf62-effb3add93a0" containerName="cleanup"
Oct 30 22:01:23.246316 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:01:23.246261    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="865d5bf1-9788-4c88-8ae6-e1a20f1efa66" containerName="cleanup"
Oct 30 22:01:23.246316 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:01:23.246293    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d70ad2e-86fa-40c7-86e3-98adba4c3913" containerName="cleanup"
Oct 30 22:01:23.250731 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:01:23.250709    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-f95pk"
Oct 30 22:01:23.289603 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:01:23.289570    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-4rghz"]
Oct 30 22:01:23.297554 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:01:23.294626    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-4rghz"
Oct 30 22:01:23.332564 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:01:23.332505    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/081e86b3-a830-4a91-a7da-c270baca6eee-workload-certs\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-f95pk\" (UID: \"081e86b3-a830-4a91-a7da-c270baca6eee\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-f95pk"
Oct 30 22:01:23.332899 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:01:23.332874    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/081e86b3-a830-4a91-a7da-c270baca6eee-credential-socket\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-f95pk\" (UID: \"081e86b3-a830-4a91-a7da-c270baca6eee\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-f95pk"
Oct 30 22:01:23.333080 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:01:23.333060    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/081e86b3-a830-4a91-a7da-c270baca6eee-istio-data\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-f95pk\" (UID: \"081e86b3-a830-4a91-a7da-c270baca6eee\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-f95pk"
Oct 30 22:01:23.333268 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:01:23.333249    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/081e86b3-a830-4a91-a7da-c270baca6eee-istio-envoy\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-f95pk\" (UID: \"081e86b3-a830-4a91-a7da-c270baca6eee\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-f95pk"
Oct 30 22:01:23.336758 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:01:23.336725    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76jqn\" (UniqueName: \"kubernetes.io/projected/081e86b3-a830-4a91-a7da-c270baca6eee-kube-api-access-76jqn\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-f95pk\" (UID: \"081e86b3-a830-4a91-a7da-c270baca6eee\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-f95pk"
Oct 30 22:01:23.337005 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:01:23.336985    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/081e86b3-a830-4a91-a7da-c270baca6eee-istio-token\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-f95pk\" (UID: \"081e86b3-a830-4a91-a7da-c270baca6eee\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-f95pk"
Oct 30 22:01:23.337175 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:01:23.337156    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/081e86b3-a830-4a91-a7da-c270baca6eee-istio-podinfo\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-f95pk\" (UID: \"081e86b3-a830-4a91-a7da-c270baca6eee\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-f95pk"
Oct 30 22:01:23.337294 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:01:23.337278    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/081e86b3-a830-4a91-a7da-c270baca6eee-workload-socket\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-f95pk\" (UID: \"081e86b3-a830-4a91-a7da-c270baca6eee\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-f95pk"
Oct 30 22:01:23.337417 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:01:23.337401    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/081e86b3-a830-4a91-a7da-c270baca6eee-istiod-ca-cert\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-f95pk\" (UID: \"081e86b3-a830-4a91-a7da-c270baca6eee\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-f95pk"
Oct 30 22:01:23.355871 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:01:23.355825    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-f95pk"]
Oct 30 22:01:23.420138 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:01:23.420101    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-4rghz"]
Oct 30 22:01:23.441493 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:01:23.441432    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/081e86b3-a830-4a91-a7da-c270baca6eee-istio-envoy\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-f95pk\" (UID: \"081e86b3-a830-4a91-a7da-c270baca6eee\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-f95pk"
Oct 30 22:01:23.441858 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:01:23.441825    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a-credential-socket\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-4rghz\" (UID: \"b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-4rghz"
Oct 30 22:01:23.442108 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:01:23.442079    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a-workload-socket\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-4rghz\" (UID: \"b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-4rghz"
Oct 30 22:01:23.442307 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:01:23.442279    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/081e86b3-a830-4a91-a7da-c270baca6eee-istio-podinfo\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-f95pk\" (UID: \"081e86b3-a830-4a91-a7da-c270baca6eee\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-f95pk"
Oct 30 22:01:23.442474 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:01:23.442447    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/081e86b3-a830-4a91-a7da-c270baca6eee-istiod-ca-cert\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-f95pk\" (UID: \"081e86b3-a830-4a91-a7da-c270baca6eee\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-f95pk"
Oct 30 22:01:23.442683 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:01:23.442659    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a-istiod-ca-cert\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-4rghz\" (UID: \"b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-4rghz"
Oct 30 22:01:23.442833 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:01:23.442816    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/081e86b3-a830-4a91-a7da-c270baca6eee-workload-certs\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-f95pk\" (UID: \"081e86b3-a830-4a91-a7da-c270baca6eee\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-f95pk"
Oct 30 22:01:23.442954 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:01:23.442937    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a-istio-token\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-4rghz\" (UID: \"b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-4rghz"
Oct 30 22:01:23.443074 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:01:23.443059    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/081e86b3-a830-4a91-a7da-c270baca6eee-credential-socket\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-f95pk\" (UID: \"081e86b3-a830-4a91-a7da-c270baca6eee\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-f95pk"
Oct 30 22:01:23.443180 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:01:23.443166    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/081e86b3-a830-4a91-a7da-c270baca6eee-istio-data\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-f95pk\" (UID: \"081e86b3-a830-4a91-a7da-c270baca6eee\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-f95pk"
Oct 30 22:01:23.443379 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:01:23.443363    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a-istio-data\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-4rghz\" (UID: \"b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-4rghz"
Oct 30 22:01:23.443512 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:01:23.443498    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76jqn\" (UniqueName: \"kubernetes.io/projected/081e86b3-a830-4a91-a7da-c270baca6eee-kube-api-access-76jqn\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-f95pk\" (UID: \"081e86b3-a830-4a91-a7da-c270baca6eee\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-f95pk"
Oct 30 22:01:23.443656 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:01:23.443639    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a-workload-certs\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-4rghz\" (UID: \"b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-4rghz"
Oct 30 22:01:23.443769 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:01:23.443753    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a-istio-podinfo\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-4rghz\" (UID: \"b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-4rghz"
Oct 30 22:01:23.443900 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:01:23.443885    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/081e86b3-a830-4a91-a7da-c270baca6eee-istio-token\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-f95pk\" (UID: \"081e86b3-a830-4a91-a7da-c270baca6eee\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-f95pk"
Oct 30 22:01:23.444021 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:01:23.444004    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a-istio-envoy\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-4rghz\" (UID: \"b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-4rghz"
Oct 30 22:01:23.444137 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:01:23.444121    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/081e86b3-a830-4a91-a7da-c270baca6eee-workload-socket\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-f95pk\" (UID: \"081e86b3-a830-4a91-a7da-c270baca6eee\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-f95pk"
Oct 30 22:01:23.444299 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:01:23.444282    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b56ms\" (UniqueName: \"kubernetes.io/projected/b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a-kube-api-access-b56ms\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-4rghz\" (UID: \"b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-4rghz"
Oct 30 22:01:23.445289 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:01:23.445269    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/081e86b3-a830-4a91-a7da-c270baca6eee-credential-socket\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-f95pk\" (UID: \"081e86b3-a830-4a91-a7da-c270baca6eee\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-f95pk"
Oct 30 22:01:23.448721 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:01:23.447267    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/081e86b3-a830-4a91-a7da-c270baca6eee-istiod-ca-cert\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-f95pk\" (UID: \"081e86b3-a830-4a91-a7da-c270baca6eee\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-f95pk"
Oct 30 22:01:23.448721 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:01:23.447549    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/081e86b3-a830-4a91-a7da-c270baca6eee-workload-certs\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-f95pk\" (UID: \"081e86b3-a830-4a91-a7da-c270baca6eee\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-f95pk"
Oct 30 22:01:23.448721 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:01:23.447840    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/081e86b3-a830-4a91-a7da-c270baca6eee-istio-data\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-f95pk\" (UID: \"081e86b3-a830-4a91-a7da-c270baca6eee\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-f95pk"
Oct 30 22:01:23.448721 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:01:23.448635    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/081e86b3-a830-4a91-a7da-c270baca6eee-workload-socket\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-f95pk\" (UID: \"081e86b3-a830-4a91-a7da-c270baca6eee\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-f95pk"
Oct 30 22:01:23.483572 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:01:23.482923    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/081e86b3-a830-4a91-a7da-c270baca6eee-istio-envoy\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-f95pk\" (UID: \"081e86b3-a830-4a91-a7da-c270baca6eee\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-f95pk"
Oct 30 22:01:23.495766 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:01:23.485580    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/081e86b3-a830-4a91-a7da-c270baca6eee-istio-podinfo\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-f95pk\" (UID: \"081e86b3-a830-4a91-a7da-c270baca6eee\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-f95pk"
Oct 30 22:01:23.538448 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:01:23.538405    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/081e86b3-a830-4a91-a7da-c270baca6eee-istio-token\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-f95pk\" (UID: \"081e86b3-a830-4a91-a7da-c270baca6eee\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-f95pk"
Oct 30 22:01:23.544580 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:01:23.541315    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76jqn\" (UniqueName: \"kubernetes.io/projected/081e86b3-a830-4a91-a7da-c270baca6eee-kube-api-access-76jqn\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-f95pk\" (UID: \"081e86b3-a830-4a91-a7da-c270baca6eee\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-f95pk"
Oct 30 22:01:23.545602 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:01:23.545570    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a-workload-certs\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-4rghz\" (UID: \"b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-4rghz"
Oct 30 22:01:23.545743 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:01:23.545620    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a-istio-podinfo\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-4rghz\" (UID: \"b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-4rghz"
Oct 30 22:01:23.545743 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:01:23.545672    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a-istio-envoy\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-4rghz\" (UID: \"b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-4rghz"
Oct 30 22:01:23.545743 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:01:23.545736    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b56ms\" (UniqueName: \"kubernetes.io/projected/b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a-kube-api-access-b56ms\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-4rghz\" (UID: \"b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-4rghz"
Oct 30 22:01:23.545983 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:01:23.545812    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a-credential-socket\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-4rghz\" (UID: \"b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-4rghz"
Oct 30 22:01:23.545983 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:01:23.545896    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a-workload-socket\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-4rghz\" (UID: \"b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-4rghz"
Oct 30 22:01:23.545983 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:01:23.545967    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a-istiod-ca-cert\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-4rghz\" (UID: \"b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-4rghz"
Oct 30 22:01:23.546142 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:01:23.545999    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a-istio-token\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-4rghz\" (UID: \"b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-4rghz"
Oct 30 22:01:23.546142 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:01:23.546117    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a-istio-data\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-4rghz\" (UID: \"b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-4rghz"
Oct 30 22:01:23.547752 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:01:23.546570    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a-istio-data\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-4rghz\" (UID: \"b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-4rghz"
Oct 30 22:01:23.547752 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:01:23.546846    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a-workload-certs\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-4rghz\" (UID: \"b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-4rghz"
Oct 30 22:01:23.549590 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:01:23.549565    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a-istiod-ca-cert\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-4rghz\" (UID: \"b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-4rghz"
Oct 30 22:01:23.550168 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:01:23.550146    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a-workload-socket\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-4rghz\" (UID: \"b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-4rghz"
Oct 30 22:01:23.550963 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:01:23.550938    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a-istio-podinfo\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-4rghz\" (UID: \"b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-4rghz"
Oct 30 22:01:23.552381 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:01:23.552361    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a-credential-socket\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-4rghz\" (UID: \"b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-4rghz"
Oct 30 22:01:23.554865 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:01:23.554694    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a-istio-envoy\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-4rghz\" (UID: \"b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-4rghz"
Oct 30 22:01:23.576863 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:01:23.576835    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-f95pk"
Oct 30 22:01:23.603756 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:01:23.603731    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a-istio-token\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-4rghz\" (UID: \"b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-4rghz"
Oct 30 22:01:23.605400 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:01:23.605378    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b56ms\" (UniqueName: \"kubernetes.io/projected/b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a-kube-api-access-b56ms\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-4rghz\" (UID: \"b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-4rghz"
Oct 30 22:01:23.616011 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:01:23.615981    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-4rghz"
Oct 30 22:01:24.009065 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:01:24.009027    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-f95pk"]
Oct 30 22:01:24.012961 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:01:24.012936    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-4rghz"]
Oct 30 22:01:24.013298 ip-10-0-130-60 kubenswrapper[5061]: W1030 22:01:24.013255    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod081e86b3_a830_4a91_a7da_c270baca6eee.slice/crio-be72527efbd5e8c7d6e2071db97dff9a6010498d3e2432ff89b1ba24302a46aa WatchSource:0}: Error finding container be72527efbd5e8c7d6e2071db97dff9a6010498d3e2432ff89b1ba24302a46aa: Status 404 returned error can't find the container with id be72527efbd5e8c7d6e2071db97dff9a6010498d3e2432ff89b1ba24302a46aa
Oct 30 22:01:24.015950 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:01:24.015920    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:01:24.016018 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:01:24.015990    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:01:24.016059 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:01:24.016019    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:01:24.053908 ip-10-0-130-60 kubenswrapper[5061]: W1030 22:01:24.053881    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3ba1489_f9d8_4325_a1ac_65bc83ce4c0a.slice/crio-a00343b7f1e0db39817bbda59a5da5b7cc895484a454b254942cce3374534d47 WatchSource:0}: Error finding container a00343b7f1e0db39817bbda59a5da5b7cc895484a454b254942cce3374534d47: Status 404 returned error can't find the container with id a00343b7f1e0db39817bbda59a5da5b7cc895484a454b254942cce3374534d47
Oct 30 22:01:24.056348 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:01:24.056315    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:01:24.056424 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:01:24.056384    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:01:24.056424 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:01:24.056414    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:01:24.682393 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:01:24.682350    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 22:01:24.769599 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:01:24.769562    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-f95pk" event={"ID":"081e86b3-a830-4a91-a7da-c270baca6eee","Type":"ContainerStarted","Data":"7e00c8d518881ac6440d33fac0fff34bc736724b28ddab555df35b72d0214c02"}
Oct 30 22:01:24.769770 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:01:24.769621    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-f95pk" event={"ID":"081e86b3-a830-4a91-a7da-c270baca6eee","Type":"ContainerStarted","Data":"be72527efbd5e8c7d6e2071db97dff9a6010498d3e2432ff89b1ba24302a46aa"}
Oct 30 22:01:24.771829 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:01:24.771802    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-4rghz" event={"ID":"b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a","Type":"ContainerStarted","Data":"d85286201179d3d2462df7f6603001b24a7185aa408d0ed8c143a442a4c81d0c"}
Oct 30 22:01:24.771982 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:01:24.771849    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-4rghz" event={"ID":"b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a","Type":"ContainerStarted","Data":"a00343b7f1e0db39817bbda59a5da5b7cc895484a454b254942cce3374534d47"}
Oct 30 22:01:24.884291 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:01:24.884235    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-f95pk" podStartSLOduration=1.884219691 podStartE2EDuration="1.884219691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:01:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:01:24.882594652 +0000 UTC m=+1138.495364409" watchObservedRunningTime="2026-10-30 22:01:24.884219691 +0000 UTC m=+1138.496989449"
Oct 30 22:01:24.997275 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:01:24.997162    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-4rghz" podStartSLOduration=1.9971453590000001 podStartE2EDuration="1.997145359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:01:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:01:24.994813359 +0000 UTC m=+1138.607583117" watchObservedRunningTime="2026-10-30 22:01:24.997145359 +0000 UTC m=+1138.609915117"
Oct 30 22:01:25.577282 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:01:25.577244    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-f95pk"
Oct 30 22:01:25.578957 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:01:25.578934    5061 patch_prober.go:29] interesting pod/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-f95pk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Oct 30 22:01:25.579032 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:01:25.578983    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-f95pk" podUID="081e86b3-a830-4a91-a7da-c270baca6eee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Oct 30 22:01:25.633552 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:01:25.632864    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-4rghz"
Oct 30 22:01:25.633552 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:01:25.633025    5061 patch_prober.go:29] interesting pod/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-4rghz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Oct 30 22:01:25.633552 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:01:25.633068    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-4rghz" podUID="b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Oct 30 22:01:26.577516 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:01:26.577485    5061 patch_prober.go:29] interesting pod/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-f95pk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Oct 30 22:01:26.577939 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:01:26.577555    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-f95pk" podUID="081e86b3-a830-4a91-a7da-c270baca6eee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Oct 30 22:01:26.632553 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:01:26.630056    5061 patch_prober.go:29] interesting pod/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-4rghz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Oct 30 22:01:26.632553 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:01:26.630133    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-4rghz" podUID="b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Oct 30 22:01:27.577843 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:01:27.577802    5061 patch_prober.go:29] interesting pod/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-f95pk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Oct 30 22:01:27.578238 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:01:27.577857    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-f95pk" podUID="081e86b3-a830-4a91-a7da-c270baca6eee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Oct 30 22:01:27.641553 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:01:27.640670    5061 patch_prober.go:29] interesting pod/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-4rghz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Oct 30 22:01:27.641553 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:01:27.640743    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-4rghz" podUID="b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Oct 30 22:01:28.578124 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:01:28.578085    5061 patch_prober.go:29] interesting pod/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-f95pk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Oct 30 22:01:28.578570 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:01:28.578139    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-f95pk" podUID="081e86b3-a830-4a91-a7da-c270baca6eee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Oct 30 22:01:28.623555 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:01:28.620999    5061 patch_prober.go:29] interesting pod/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-4rghz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Oct 30 22:01:28.623555 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:01:28.621067    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-4rghz" podUID="b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Oct 30 22:01:29.577565 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:01:29.577512    5061 patch_prober.go:29] interesting pod/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-f95pk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Oct 30 22:01:29.577746 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:01:29.577591    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-f95pk" podUID="081e86b3-a830-4a91-a7da-c270baca6eee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Oct 30 22:01:29.629553 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:01:29.628909    5061 patch_prober.go:29] interesting pod/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-4rghz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Oct 30 22:01:29.629553 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:01:29.628983    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-4rghz" podUID="b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Oct 30 22:01:30.577509 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:01:30.577462    5061 patch_prober.go:29] interesting pod/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-f95pk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Oct 30 22:01:30.577758 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:01:30.577523    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-f95pk" podUID="081e86b3-a830-4a91-a7da-c270baca6eee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Oct 30 22:01:30.616604 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:01:30.616558    5061 patch_prober.go:29] interesting pod/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-4rghz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Oct 30 22:01:30.616816 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:01:30.616625    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-4rghz" podUID="b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Oct 30 22:01:31.581900 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:01:31.581869    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-f95pk"
Oct 30 22:01:31.582333 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:01:31.582262    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-f95pk"
Oct 30 22:01:31.582863 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:01:31.582840    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-f95pk"
Oct 30 22:01:31.621095 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:01:31.621064    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-4rghz"
Oct 30 22:01:31.621361 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:01:31.621340    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-4rghz"
Oct 30 22:01:31.622118 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:01:31.622102    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-4rghz"
Oct 30 22:01:54.668053 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:01:54.668012    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 22:02:13.511630 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:13.511594    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-f776cb/distinct-2-w3-f776cb-kserve-7b4d4cc954-d88pf"]
Oct 30 22:02:13.516866 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:13.516843    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-f776cb/distinct-2-w3-f776cb-kserve-7b4d4cc954-d88pf"
Oct 30 22:02:13.601442 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:13.601404    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b184062-7fac-43d4-ba5f-21567df85da3-dshm\") pod \"distinct-2-w3-f776cb-kserve-7b4d4cc954-d88pf\" (UID: \"5b184062-7fac-43d4-ba5f-21567df85da3\") " pod="e2e-models-e2e-worker-w3-f776cb/distinct-2-w3-f776cb-kserve-7b4d4cc954-d88pf"
Oct 30 22:02:13.601694 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:13.601481    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksz9n\" (UniqueName: \"kubernetes.io/projected/5b184062-7fac-43d4-ba5f-21567df85da3-kube-api-access-ksz9n\") pod \"distinct-2-w3-f776cb-kserve-7b4d4cc954-d88pf\" (UID: \"5b184062-7fac-43d4-ba5f-21567df85da3\") " pod="e2e-models-e2e-worker-w3-f776cb/distinct-2-w3-f776cb-kserve-7b4d4cc954-d88pf"
Oct 30 22:02:13.601694 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:13.601663    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b184062-7fac-43d4-ba5f-21567df85da3-model-cache\") pod \"distinct-2-w3-f776cb-kserve-7b4d4cc954-d88pf\" (UID: \"5b184062-7fac-43d4-ba5f-21567df85da3\") " pod="e2e-models-e2e-worker-w3-f776cb/distinct-2-w3-f776cb-kserve-7b4d4cc954-d88pf"
Oct 30 22:02:13.601839 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:13.601797    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b184062-7fac-43d4-ba5f-21567df85da3-tls-certs\") pod \"distinct-2-w3-f776cb-kserve-7b4d4cc954-d88pf\" (UID: \"5b184062-7fac-43d4-ba5f-21567df85da3\") " pod="e2e-models-e2e-worker-w3-f776cb/distinct-2-w3-f776cb-kserve-7b4d4cc954-d88pf"
Oct 30 22:02:13.601919 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:13.601902    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b184062-7fac-43d4-ba5f-21567df85da3-tmp-dir\") pod \"distinct-2-w3-f776cb-kserve-7b4d4cc954-d88pf\" (UID: \"5b184062-7fac-43d4-ba5f-21567df85da3\") " pod="e2e-models-e2e-worker-w3-f776cb/distinct-2-w3-f776cb-kserve-7b4d4cc954-d88pf"
Oct 30 22:02:13.601982 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:13.601934    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b184062-7fac-43d4-ba5f-21567df85da3-home\") pod \"distinct-2-w3-f776cb-kserve-7b4d4cc954-d88pf\" (UID: \"5b184062-7fac-43d4-ba5f-21567df85da3\") " pod="e2e-models-e2e-worker-w3-f776cb/distinct-2-w3-f776cb-kserve-7b4d4cc954-d88pf"
Oct 30 22:02:13.614867 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:13.614834    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-f776cb\"/\"openshift-service-ca.crt\""
Oct 30 22:02:13.614867 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:13.614834    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-f776cb\"/\"default-dockercfg-xf7tg\""
Oct 30 22:02:13.615103 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:13.614917    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-f776cb\"/\"distinct-2-w3-f776cb-kserve-self-signed-certs\""
Oct 30 22:02:13.615103 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:13.614922    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-f776cb\"/\"kube-root-ca.crt\""
Oct 30 22:02:13.618952 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:13.618912    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-f776cb/distinct-2-w3-f776cb-kserve-7b4d4cc954-d88pf"]
Oct 30 22:02:13.702945 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:13.702911    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b184062-7fac-43d4-ba5f-21567df85da3-dshm\") pod \"distinct-2-w3-f776cb-kserve-7b4d4cc954-d88pf\" (UID: \"5b184062-7fac-43d4-ba5f-21567df85da3\") " pod="e2e-models-e2e-worker-w3-f776cb/distinct-2-w3-f776cb-kserve-7b4d4cc954-d88pf"
Oct 30 22:02:13.703116 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:13.703002    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksz9n\" (UniqueName: \"kubernetes.io/projected/5b184062-7fac-43d4-ba5f-21567df85da3-kube-api-access-ksz9n\") pod \"distinct-2-w3-f776cb-kserve-7b4d4cc954-d88pf\" (UID: \"5b184062-7fac-43d4-ba5f-21567df85da3\") " pod="e2e-models-e2e-worker-w3-f776cb/distinct-2-w3-f776cb-kserve-7b4d4cc954-d88pf"
Oct 30 22:02:13.703116 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:13.703041    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b184062-7fac-43d4-ba5f-21567df85da3-model-cache\") pod \"distinct-2-w3-f776cb-kserve-7b4d4cc954-d88pf\" (UID: \"5b184062-7fac-43d4-ba5f-21567df85da3\") " pod="e2e-models-e2e-worker-w3-f776cb/distinct-2-w3-f776cb-kserve-7b4d4cc954-d88pf"
Oct 30 22:02:13.703208 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:13.703114    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b184062-7fac-43d4-ba5f-21567df85da3-tls-certs\") pod \"distinct-2-w3-f776cb-kserve-7b4d4cc954-d88pf\" (UID: \"5b184062-7fac-43d4-ba5f-21567df85da3\") " pod="e2e-models-e2e-worker-w3-f776cb/distinct-2-w3-f776cb-kserve-7b4d4cc954-d88pf"
Oct 30 22:02:13.703208 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:13.703168    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b184062-7fac-43d4-ba5f-21567df85da3-tmp-dir\") pod \"distinct-2-w3-f776cb-kserve-7b4d4cc954-d88pf\" (UID: \"5b184062-7fac-43d4-ba5f-21567df85da3\") " pod="e2e-models-e2e-worker-w3-f776cb/distinct-2-w3-f776cb-kserve-7b4d4cc954-d88pf"
Oct 30 22:02:13.703208 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:13.703185    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b184062-7fac-43d4-ba5f-21567df85da3-home\") pod \"distinct-2-w3-f776cb-kserve-7b4d4cc954-d88pf\" (UID: \"5b184062-7fac-43d4-ba5f-21567df85da3\") " pod="e2e-models-e2e-worker-w3-f776cb/distinct-2-w3-f776cb-kserve-7b4d4cc954-d88pf"
Oct 30 22:02:13.703506 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:13.703481    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b184062-7fac-43d4-ba5f-21567df85da3-model-cache\") pod \"distinct-2-w3-f776cb-kserve-7b4d4cc954-d88pf\" (UID: \"5b184062-7fac-43d4-ba5f-21567df85da3\") " pod="e2e-models-e2e-worker-w3-f776cb/distinct-2-w3-f776cb-kserve-7b4d4cc954-d88pf"
Oct 30 22:02:13.703672 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:13.703643    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b184062-7fac-43d4-ba5f-21567df85da3-tmp-dir\") pod \"distinct-2-w3-f776cb-kserve-7b4d4cc954-d88pf\" (UID: \"5b184062-7fac-43d4-ba5f-21567df85da3\") " pod="e2e-models-e2e-worker-w3-f776cb/distinct-2-w3-f776cb-kserve-7b4d4cc954-d88pf"
Oct 30 22:02:13.703809 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:13.703658    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b184062-7fac-43d4-ba5f-21567df85da3-home\") pod \"distinct-2-w3-f776cb-kserve-7b4d4cc954-d88pf\" (UID: \"5b184062-7fac-43d4-ba5f-21567df85da3\") " pod="e2e-models-e2e-worker-w3-f776cb/distinct-2-w3-f776cb-kserve-7b4d4cc954-d88pf"
Oct 30 22:02:13.705466 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:13.705443    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b184062-7fac-43d4-ba5f-21567df85da3-dshm\") pod \"distinct-2-w3-f776cb-kserve-7b4d4cc954-d88pf\" (UID: \"5b184062-7fac-43d4-ba5f-21567df85da3\") " pod="e2e-models-e2e-worker-w3-f776cb/distinct-2-w3-f776cb-kserve-7b4d4cc954-d88pf"
Oct 30 22:02:13.705658 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:13.705640    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b184062-7fac-43d4-ba5f-21567df85da3-tls-certs\") pod \"distinct-2-w3-f776cb-kserve-7b4d4cc954-d88pf\" (UID: \"5b184062-7fac-43d4-ba5f-21567df85da3\") " pod="e2e-models-e2e-worker-w3-f776cb/distinct-2-w3-f776cb-kserve-7b4d4cc954-d88pf"
Oct 30 22:02:13.758067 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:13.758034    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksz9n\" (UniqueName: \"kubernetes.io/projected/5b184062-7fac-43d4-ba5f-21567df85da3-kube-api-access-ksz9n\") pod \"distinct-2-w3-f776cb-kserve-7b4d4cc954-d88pf\" (UID: \"5b184062-7fac-43d4-ba5f-21567df85da3\") " pod="e2e-models-e2e-worker-w3-f776cb/distinct-2-w3-f776cb-kserve-7b4d4cc954-d88pf"
Oct 30 22:02:13.830449 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:13.830412    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-f776cb/distinct-2-w3-f776cb-kserve-7b4d4cc954-d88pf"
Oct 30 22:02:14.235951 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:14.235724    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-f776cb/distinct-2-w3-f776cb-kserve-7b4d4cc954-d88pf"]
Oct 30 22:02:14.239951 ip-10-0-130-60 kubenswrapper[5061]: W1030 22:02:14.239923    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5b184062_7fac_43d4_ba5f_21567df85da3.slice/crio-2eba29a30db6219883ca4d99f9b4dddaf8388c49dbe112cedecc71e5f98af3f6 WatchSource:0}: Error finding container 2eba29a30db6219883ca4d99f9b4dddaf8388c49dbe112cedecc71e5f98af3f6: Status 404 returned error can't find the container with id 2eba29a30db6219883ca4d99f9b4dddaf8388c49dbe112cedecc71e5f98af3f6
Oct 30 22:02:15.183771 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:15.183735    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-f776cb/distinct-2-w3-f776cb-kserve-7b4d4cc954-d88pf" event={"ID":"5b184062-7fac-43d4-ba5f-21567df85da3","Type":"ContainerStarted","Data":"4bc79af284930e2e37a47baa13155977e23227bd5f18297f0987384964813230"}
Oct 30 22:02:15.184213 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:15.183782    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-f776cb/distinct-2-w3-f776cb-kserve-7b4d4cc954-d88pf" event={"ID":"5b184062-7fac-43d4-ba5f-21567df85da3","Type":"ContainerStarted","Data":"2eba29a30db6219883ca4d99f9b4dddaf8388c49dbe112cedecc71e5f98af3f6"}
Oct 30 22:02:15.299033 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:15.298972    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-f776cb/distinct-2-w3-f776cb-kserve-7b4d4cc954-d88pf" podStartSLOduration=2.298954463 podStartE2EDuration="2.298954463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:02:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:02:15.295348592 +0000 UTC m=+1188.908118350" watchObservedRunningTime="2026-10-30 22:02:15.298954463 +0000 UTC m=+1188.911724220"
Oct 30 22:02:15.830946 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:15.830909    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f776cb/distinct-2-w3-f776cb-kserve-7b4d4cc954-d88pf"
Oct 30 22:02:17.339952 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:17.337643    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-17d809/premium-w4-17d809-kserve-7c9bdbbcbc-5mtmm" podUID="1e0eca81-7db1-482b-8dd6-5b28fb89e010" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:02:17.411344 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:17.409553    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2d1f91/simulator-w0-2d1f91-kserve-7699765c74-jwtkb" podUID="44b7d6f6-32d8-4ef4-95bc-eef604fa6b5b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 22:02:17.491552 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:17.490713    5061 patch_prober.go:29] interesting pod/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5zvwr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:02:17.491552 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:17.490847    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5zvwr" podUID="3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:02:17.524552 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:17.523613    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a07620/simulator-w1-a07620-kserve-55b44674f4-4z9lk" podUID="efe7c063-fb0e-4679-8a7c-13f2a0abaa74" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:02:17.561570 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:17.550355    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-14af5c/distinct-2-w6-14af5c-kserve-58d554bdb6-r5r6j" podUID="990f4e94-708e-45f3-b2ad-01cd50b3257b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:02:17.604854 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:17.604745    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-14af5c/simulator-w6-14af5c-kserve-85b8bd96d7-wlcb9" podUID="39766c7d-5f1d-42db-8920-c0321cd12c99" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:02:17.798996 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:17.798932    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-14af5c/distinct-w6-14af5c-kserve-7c7cb9d69c-hljl9" podUID="1f8d9486-fb2f-491a-a8d7-c23c21bd116a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:02:17.858609 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:17.834069    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-17d809/simulator-w4-17d809-kserve-78d67b6f4-7hsvb" podUID="e7337cf8-4f59-4afb-9b50-6b6c9fa4a773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:02:17.881552 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:17.863114    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2d1f91/premium-w0-2d1f91-kserve-6965b558cf-59lgw" podUID="fb84233d-25e0-48bc-8899-c3a3cb3cfeb9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:02:18.089549 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:18.089082    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-14af5c/premium-w6-14af5c-kserve-6549647f4-f4gwg" podUID="9b42ee01-c60e-462c-a0e0-b04a52014db6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:02:18.150099 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:18.150000    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-blrrb" podUID="cb4b1ed5-2192-4877-8be7-ca82af831af5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:02:18.163583 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:18.153423    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-17d809/distinct-w4-17d809-kserve-6b8f5c48cd-lhrwp" podUID="ef5e8691-42e1-471b-ba7d-90881028eca7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:02:19.738939 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:19.738847    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2197e210-58t8k"]
Oct 30 22:02:19.745080 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:19.745050    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2197e210-58t8k"
Oct 30 22:02:19.795114 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:19.795078    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:02:19.795114 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:19.795087    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gdmfg\""
Oct 30 22:02:19.797599 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:19.797568    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2197e210-58t8k"]
Oct 30 22:02:19.817202 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:19.817155    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8002160f-c9be-4542-9147-860ddb5652ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2197e210-58t8k\" (UID: \"8002160f-c9be-4542-9147-860ddb5652ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2197e210-58t8k"
Oct 30 22:02:19.817202 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:19.817209    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8002160f-c9be-4542-9147-860ddb5652ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2197e210-58t8k\" (UID: \"8002160f-c9be-4542-9147-860ddb5652ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2197e210-58t8k"
Oct 30 22:02:19.817455 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:19.817422    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hz2f4\" (UniqueName: \"kubernetes.io/projected/8002160f-c9be-4542-9147-860ddb5652ca-kube-api-access-hz2f4\") pod \"maas-api-revoke-sub-2197e210-58t8k\" (UID: \"8002160f-c9be-4542-9147-860ddb5652ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2197e210-58t8k"
Oct 30 22:02:19.918894 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:19.918858    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8002160f-c9be-4542-9147-860ddb5652ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2197e210-58t8k\" (UID: \"8002160f-c9be-4542-9147-860ddb5652ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2197e210-58t8k"
Oct 30 22:02:19.918894 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:19.918894    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8002160f-c9be-4542-9147-860ddb5652ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2197e210-58t8k\" (UID: \"8002160f-c9be-4542-9147-860ddb5652ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2197e210-58t8k"
Oct 30 22:02:19.919159 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:19.918994    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hz2f4\" (UniqueName: \"kubernetes.io/projected/8002160f-c9be-4542-9147-860ddb5652ca-kube-api-access-hz2f4\") pod \"maas-api-revoke-sub-2197e210-58t8k\" (UID: \"8002160f-c9be-4542-9147-860ddb5652ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2197e210-58t8k"
Oct 30 22:02:19.919790 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:19.919749    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8002160f-c9be-4542-9147-860ddb5652ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2197e210-58t8k\" (UID: \"8002160f-c9be-4542-9147-860ddb5652ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2197e210-58t8k"
Oct 30 22:02:19.921317 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:19.921288    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8002160f-c9be-4542-9147-860ddb5652ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2197e210-58t8k\" (UID: \"8002160f-c9be-4542-9147-860ddb5652ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2197e210-58t8k"
Oct 30 22:02:19.977931 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:19.977900    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hz2f4\" (UniqueName: \"kubernetes.io/projected/8002160f-c9be-4542-9147-860ddb5652ca-kube-api-access-hz2f4\") pod \"maas-api-revoke-sub-2197e210-58t8k\" (UID: \"8002160f-c9be-4542-9147-860ddb5652ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2197e210-58t8k"
Oct 30 22:02:20.062305 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:20.062268    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2197e210-58t8k"
Oct 30 22:02:20.468812 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:20.468657    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2197e210-58t8k"]
Oct 30 22:02:20.473201 ip-10-0-130-60 kubenswrapper[5061]: W1030 22:02:20.473175    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8002160f_c9be_4542_9147_860ddb5652ca.slice/crio-080510bbf625f2facb1bb0585d120d002eebbf21fdbf3112e94576d781a195b8 WatchSource:0}: Error finding container 080510bbf625f2facb1bb0585d120d002eebbf21fdbf3112e94576d781a195b8: Status 404 returned error can't find the container with id 080510bbf625f2facb1bb0585d120d002eebbf21fdbf3112e94576d781a195b8
Oct 30 22:02:21.248403 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:21.248367    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2197e210-58t8k" event={"ID":"8002160f-c9be-4542-9147-860ddb5652ca","Type":"ContainerStarted","Data":"42ef4afa9fddcbc85c0000a3c7f2e3a496f040fae73f285cdfc10a484e47386c"}
Oct 30 22:02:21.248868 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:21.248424    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2197e210-58t8k" event={"ID":"8002160f-c9be-4542-9147-860ddb5652ca","Type":"ContainerStarted","Data":"080510bbf625f2facb1bb0585d120d002eebbf21fdbf3112e94576d781a195b8"}
Oct 30 22:02:21.361192 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:21.361136    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2197e210-58t8k" podStartSLOduration=2.361122496 podStartE2EDuration="2.361122496s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:02:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:02:21.360643636 +0000 UTC m=+1194.973413391" watchObservedRunningTime="2026-10-30 22:02:21.361122496 +0000 UTC m=+1194.973892253"
Oct 30 22:02:22.266991 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:22.266933    5061 generic.go:356] "Generic (PLEG): container finished" podID="8002160f-c9be-4542-9147-860ddb5652ca" containerID="42ef4afa9fddcbc85c0000a3c7f2e3a496f040fae73f285cdfc10a484e47386c" exitCode=0
Oct 30 22:02:22.267487 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:22.267056    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2197e210-58t8k" event={"ID":"8002160f-c9be-4542-9147-860ddb5652ca","Type":"ContainerDied","Data":"42ef4afa9fddcbc85c0000a3c7f2e3a496f040fae73f285cdfc10a484e47386c"}
Oct 30 22:02:23.428994 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:23.428964    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2197e210-58t8k"
Oct 30 22:02:23.571993 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:23.571956    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8002160f-c9be-4542-9147-860ddb5652ca-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8002160f-c9be-4542-9147-860ddb5652ca-maas-api-xks-ca\") pod \"8002160f-c9be-4542-9147-860ddb5652ca\" (UID: \"8002160f-c9be-4542-9147-860ddb5652ca\") "
Oct 30 22:02:23.572177 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:23.572077    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8002160f-c9be-4542-9147-860ddb5652ca-kube-api-access-hz2f4\" (UniqueName: \"kubernetes.io/projected/8002160f-c9be-4542-9147-860ddb5652ca-kube-api-access-hz2f4\") pod \"8002160f-c9be-4542-9147-860ddb5652ca\" (UID: \"8002160f-c9be-4542-9147-860ddb5652ca\") "
Oct 30 22:02:23.572244 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:23.572203    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8002160f-c9be-4542-9147-860ddb5652ca-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8002160f-c9be-4542-9147-860ddb5652ca-maas-api-openshift-ca\") pod \"8002160f-c9be-4542-9147-860ddb5652ca\" (UID: \"8002160f-c9be-4542-9147-860ddb5652ca\") "
Oct 30 22:02:23.572631 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:23.572602    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8002160f-c9be-4542-9147-860ddb5652ca-maas-api-openshift-ca" pod "8002160f-c9be-4542-9147-860ddb5652ca" (UID: "8002160f-c9be-4542-9147-860ddb5652ca"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:02:23.572835 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:23.572817    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8002160f-c9be-4542-9147-860ddb5652ca-maas-api-openshift-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:02:23.574020 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:23.573996    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8002160f-c9be-4542-9147-860ddb5652ca-maas-api-xks-ca" pod "8002160f-c9be-4542-9147-860ddb5652ca" (UID: "8002160f-c9be-4542-9147-860ddb5652ca"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:02:23.574184 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:23.574166    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8002160f-c9be-4542-9147-860ddb5652ca-kube-api-access-hz2f4" pod "8002160f-c9be-4542-9147-860ddb5652ca" (UID: "8002160f-c9be-4542-9147-860ddb5652ca"). InnerVolumeSpecName "kube-api-access-hz2f4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:02:23.673789 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:23.673757    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8002160f-c9be-4542-9147-860ddb5652ca-maas-api-xks-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:02:23.673789 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:23.673788    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hz2f4\" (UniqueName: \"kubernetes.io/projected/8002160f-c9be-4542-9147-860ddb5652ca-kube-api-access-hz2f4\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:02:23.831383 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:23.831282    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-f776cb/distinct-2-w3-f776cb-kserve-7b4d4cc954-d88pf"
Oct 30 22:02:23.844080 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:23.844050    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-f776cb/distinct-2-w3-f776cb-kserve-7b4d4cc954-d88pf"
Oct 30 22:02:24.284548 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:24.284455    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2197e210-58t8k"
Oct 30 22:02:24.284548 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:24.284496    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2197e210-58t8k" event={"ID":"8002160f-c9be-4542-9147-860ddb5652ca","Type":"ContainerDied","Data":"080510bbf625f2facb1bb0585d120d002eebbf21fdbf3112e94576d781a195b8"}
Oct 30 22:02:24.284771 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:24.284563    5061 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="080510bbf625f2facb1bb0585d120d002eebbf21fdbf3112e94576d781a195b8"
Oct 30 22:02:24.299815 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:24.299790    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f776cb/distinct-2-w3-f776cb-kserve-7b4d4cc954-d88pf"
Oct 30 22:02:24.667358 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:02:24.667320    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 22:02:26.691574 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:26.691546    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-sclrt_5e74071e-0462-42b5-a05c-d90bb638478a/console-operator/2.log"
Oct 30 22:02:26.693268 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:26.693242    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-sclrt_5e74071e-0462-42b5-a05c-d90bb638478a/console-operator/2.log"
Oct 30 22:02:27.050299 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:27.049968    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a761260c-hqtc6"]
Oct 30 22:02:27.093997 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:27.093964    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a761260c-hqtc6"]
Oct 30 22:02:28.530148 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:28.530113    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f98000ef-7846-4621-973a-ebb69d90f775" path="/var/lib/kubelet/pods/f98000ef-7846-4621-973a-ebb69d90f775/volumes"
Oct 30 22:02:38.450786 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:38.450697    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-kdmlq"]
Oct 30 22:02:38.451543 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:38.451508    5061 cpu_manager.go:414] "containerMap: removing container" podUID="8002160f-c9be-4542-9147-860ddb5652ca" containerName="revoke-keys"
Oct 30 22:02:38.451543 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:38.451544    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8002160f-c9be-4542-9147-860ddb5652ca" containerName="revoke-keys"
Oct 30 22:02:38.451735 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:38.451682    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="8002160f-c9be-4542-9147-860ddb5652ca" containerName="revoke-keys"
Oct 30 22:02:38.456278 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:38.456261    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-kdmlq"
Oct 30 22:02:38.556511 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:38.556478    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-kdmlq"]
Oct 30 22:02:38.577051 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:38.576732    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86ngg\" (UniqueName: \"kubernetes.io/projected/1aec235a-1d74-4044-aa3d-bd87a6a2d86f-kube-api-access-86ngg\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-kdmlq\" (UID: \"1aec235a-1d74-4044-aa3d-bd87a6a2d86f\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-kdmlq"
Oct 30 22:02:38.577051 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:38.576808    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1aec235a-1d74-4044-aa3d-bd87a6a2d86f-istio-envoy\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-kdmlq\" (UID: \"1aec235a-1d74-4044-aa3d-bd87a6a2d86f\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-kdmlq"
Oct 30 22:02:38.577051 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:38.576922    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1aec235a-1d74-4044-aa3d-bd87a6a2d86f-istio-token\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-kdmlq\" (UID: \"1aec235a-1d74-4044-aa3d-bd87a6a2d86f\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-kdmlq"
Oct 30 22:02:38.577051 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:38.576971    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1aec235a-1d74-4044-aa3d-bd87a6a2d86f-workload-certs\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-kdmlq\" (UID: \"1aec235a-1d74-4044-aa3d-bd87a6a2d86f\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-kdmlq"
Oct 30 22:02:38.577051 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:38.577009    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1aec235a-1d74-4044-aa3d-bd87a6a2d86f-istiod-ca-cert\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-kdmlq\" (UID: \"1aec235a-1d74-4044-aa3d-bd87a6a2d86f\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-kdmlq"
Oct 30 22:02:38.582553 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:38.578521    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1aec235a-1d74-4044-aa3d-bd87a6a2d86f-istio-podinfo\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-kdmlq\" (UID: \"1aec235a-1d74-4044-aa3d-bd87a6a2d86f\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-kdmlq"
Oct 30 22:02:38.582553 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:38.578810    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aec235a-1d74-4044-aa3d-bd87a6a2d86f-credential-socket\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-kdmlq\" (UID: \"1aec235a-1d74-4044-aa3d-bd87a6a2d86f\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-kdmlq"
Oct 30 22:02:38.582553 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:38.578939    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1aec235a-1d74-4044-aa3d-bd87a6a2d86f-istio-data\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-kdmlq\" (UID: \"1aec235a-1d74-4044-aa3d-bd87a6a2d86f\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-kdmlq"
Oct 30 22:02:38.582553 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:38.579340    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aec235a-1d74-4044-aa3d-bd87a6a2d86f-workload-socket\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-kdmlq\" (UID: \"1aec235a-1d74-4044-aa3d-bd87a6a2d86f\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-kdmlq"
Oct 30 22:02:38.681964 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:38.681915    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aec235a-1d74-4044-aa3d-bd87a6a2d86f-credential-socket\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-kdmlq\" (UID: \"1aec235a-1d74-4044-aa3d-bd87a6a2d86f\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-kdmlq"
Oct 30 22:02:38.682198 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:38.682007    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1aec235a-1d74-4044-aa3d-bd87a6a2d86f-istio-data\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-kdmlq\" (UID: \"1aec235a-1d74-4044-aa3d-bd87a6a2d86f\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-kdmlq"
Oct 30 22:02:38.682198 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:38.682075    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aec235a-1d74-4044-aa3d-bd87a6a2d86f-workload-socket\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-kdmlq\" (UID: \"1aec235a-1d74-4044-aa3d-bd87a6a2d86f\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-kdmlq"
Oct 30 22:02:38.682198 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:38.682139    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86ngg\" (UniqueName: \"kubernetes.io/projected/1aec235a-1d74-4044-aa3d-bd87a6a2d86f-kube-api-access-86ngg\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-kdmlq\" (UID: \"1aec235a-1d74-4044-aa3d-bd87a6a2d86f\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-kdmlq"
Oct 30 22:02:38.682198 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:38.682167    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1aec235a-1d74-4044-aa3d-bd87a6a2d86f-istio-envoy\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-kdmlq\" (UID: \"1aec235a-1d74-4044-aa3d-bd87a6a2d86f\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-kdmlq"
Oct 30 22:02:38.682453 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:38.682243    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1aec235a-1d74-4044-aa3d-bd87a6a2d86f-istio-token\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-kdmlq\" (UID: \"1aec235a-1d74-4044-aa3d-bd87a6a2d86f\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-kdmlq"
Oct 30 22:02:38.682453 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:38.682387    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aec235a-1d74-4044-aa3d-bd87a6a2d86f-credential-socket\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-kdmlq\" (UID: \"1aec235a-1d74-4044-aa3d-bd87a6a2d86f\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-kdmlq"
Oct 30 22:02:38.682595 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:38.682459    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aec235a-1d74-4044-aa3d-bd87a6a2d86f-workload-socket\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-kdmlq\" (UID: \"1aec235a-1d74-4044-aa3d-bd87a6a2d86f\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-kdmlq"
Oct 30 22:02:38.682595 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:38.682490    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1aec235a-1d74-4044-aa3d-bd87a6a2d86f-istio-data\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-kdmlq\" (UID: \"1aec235a-1d74-4044-aa3d-bd87a6a2d86f\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-kdmlq"
Oct 30 22:02:38.682710 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:38.682619    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1aec235a-1d74-4044-aa3d-bd87a6a2d86f-workload-certs\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-kdmlq\" (UID: \"1aec235a-1d74-4044-aa3d-bd87a6a2d86f\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-kdmlq"
Oct 30 22:02:38.682710 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:38.682672    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1aec235a-1d74-4044-aa3d-bd87a6a2d86f-istiod-ca-cert\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-kdmlq\" (UID: \"1aec235a-1d74-4044-aa3d-bd87a6a2d86f\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-kdmlq"
Oct 30 22:02:38.682819 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:38.682717    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1aec235a-1d74-4044-aa3d-bd87a6a2d86f-istio-podinfo\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-kdmlq\" (UID: \"1aec235a-1d74-4044-aa3d-bd87a6a2d86f\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-kdmlq"
Oct 30 22:02:38.682819 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:38.682726    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1aec235a-1d74-4044-aa3d-bd87a6a2d86f-workload-certs\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-kdmlq\" (UID: \"1aec235a-1d74-4044-aa3d-bd87a6a2d86f\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-kdmlq"
Oct 30 22:02:38.683227 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:38.683199    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1aec235a-1d74-4044-aa3d-bd87a6a2d86f-istiod-ca-cert\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-kdmlq\" (UID: \"1aec235a-1d74-4044-aa3d-bd87a6a2d86f\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-kdmlq"
Oct 30 22:02:38.684805 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:38.684779    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1aec235a-1d74-4044-aa3d-bd87a6a2d86f-istio-envoy\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-kdmlq\" (UID: \"1aec235a-1d74-4044-aa3d-bd87a6a2d86f\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-kdmlq"
Oct 30 22:02:38.685180 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:38.685163    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1aec235a-1d74-4044-aa3d-bd87a6a2d86f-istio-podinfo\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-kdmlq\" (UID: \"1aec235a-1d74-4044-aa3d-bd87a6a2d86f\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-kdmlq"
Oct 30 22:02:38.820401 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:38.820371    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1aec235a-1d74-4044-aa3d-bd87a6a2d86f-istio-token\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-kdmlq\" (UID: \"1aec235a-1d74-4044-aa3d-bd87a6a2d86f\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-kdmlq"
Oct 30 22:02:38.820847 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:38.820825    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86ngg\" (UniqueName: \"kubernetes.io/projected/1aec235a-1d74-4044-aa3d-bd87a6a2d86f-kube-api-access-86ngg\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-kdmlq\" (UID: \"1aec235a-1d74-4044-aa3d-bd87a6a2d86f\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-kdmlq"
Oct 30 22:02:39.069985 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:39.069943    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-kdmlq"
Oct 30 22:02:39.488380 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:39.488288    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-wg5rk"]
Oct 30 22:02:39.494214 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:39.494184    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-wg5rk"
Oct 30 22:02:39.573625 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:39.573577    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-c79hw"]
Oct 30 22:02:39.587667 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:39.579035    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-c79hw"
Oct 30 22:02:39.591555 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:39.591476    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-kdmlq"]
Oct 30 22:02:39.599857 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:39.599793    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-wg5rk"]
Oct 30 22:02:39.600862 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:39.600827    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:02:39.600981 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:39.600899    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:02:39.600981 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:39.600942    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:02:39.638556 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:39.636452    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/025aea26-9742-4272-9249-2e916c03d53d-istio-envoy\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-wg5rk\" (UID: \"025aea26-9742-4272-9249-2e916c03d53d\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-wg5rk"
Oct 30 22:02:39.638556 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:39.636513    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/025aea26-9742-4272-9249-2e916c03d53d-workload-certs\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-wg5rk\" (UID: \"025aea26-9742-4272-9249-2e916c03d53d\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-wg5rk"
Oct 30 22:02:39.638556 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:39.636615    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/025aea26-9742-4272-9249-2e916c03d53d-istio-podinfo\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-wg5rk\" (UID: \"025aea26-9742-4272-9249-2e916c03d53d\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-wg5rk"
Oct 30 22:02:39.638556 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:39.636721    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/025aea26-9742-4272-9249-2e916c03d53d-workload-socket\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-wg5rk\" (UID: \"025aea26-9742-4272-9249-2e916c03d53d\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-wg5rk"
Oct 30 22:02:39.638556 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:39.636807    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/025aea26-9742-4272-9249-2e916c03d53d-istio-token\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-wg5rk\" (UID: \"025aea26-9742-4272-9249-2e916c03d53d\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-wg5rk"
Oct 30 22:02:39.638556 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:39.636858    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/025aea26-9742-4272-9249-2e916c03d53d-credential-socket\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-wg5rk\" (UID: \"025aea26-9742-4272-9249-2e916c03d53d\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-wg5rk"
Oct 30 22:02:39.638556 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:39.636967    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/025aea26-9742-4272-9249-2e916c03d53d-istio-data\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-wg5rk\" (UID: \"025aea26-9742-4272-9249-2e916c03d53d\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-wg5rk"
Oct 30 22:02:39.638556 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:39.637039    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/025aea26-9742-4272-9249-2e916c03d53d-istiod-ca-cert\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-wg5rk\" (UID: \"025aea26-9742-4272-9249-2e916c03d53d\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-wg5rk"
Oct 30 22:02:39.638556 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:39.637081    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45jmz\" (UniqueName: \"kubernetes.io/projected/025aea26-9742-4272-9249-2e916c03d53d-kube-api-access-45jmz\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-wg5rk\" (UID: \"025aea26-9742-4272-9249-2e916c03d53d\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-wg5rk"
Oct 30 22:02:39.795188 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:39.792584    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-c79hw"]
Oct 30 22:02:39.806967 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:39.798689    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfae7061-0326-46fd-901b-0399080c2094-workload-socket\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-c79hw\" (UID: \"bfae7061-0326-46fd-901b-0399080c2094\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-c79hw"
Oct 30 22:02:39.806967 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:39.798737    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/025aea26-9742-4272-9249-2e916c03d53d-istio-data\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-wg5rk\" (UID: \"025aea26-9742-4272-9249-2e916c03d53d\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-wg5rk"
Oct 30 22:02:39.806967 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:39.798802    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/025aea26-9742-4272-9249-2e916c03d53d-istiod-ca-cert\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-wg5rk\" (UID: \"025aea26-9742-4272-9249-2e916c03d53d\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-wg5rk"
Oct 30 22:02:39.806967 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:39.798836    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxlzx\" (UniqueName: \"kubernetes.io/projected/bfae7061-0326-46fd-901b-0399080c2094-kube-api-access-bxlzx\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-c79hw\" (UID: \"bfae7061-0326-46fd-901b-0399080c2094\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-c79hw"
Oct 30 22:02:39.806967 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:39.798870    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45jmz\" (UniqueName: \"kubernetes.io/projected/025aea26-9742-4272-9249-2e916c03d53d-kube-api-access-45jmz\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-wg5rk\" (UID: \"025aea26-9742-4272-9249-2e916c03d53d\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-wg5rk"
Oct 30 22:02:39.806967 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:39.798902    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfae7061-0326-46fd-901b-0399080c2094-credential-socket\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-c79hw\" (UID: \"bfae7061-0326-46fd-901b-0399080c2094\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-c79hw"
Oct 30 22:02:39.806967 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:39.798960    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfae7061-0326-46fd-901b-0399080c2094-workload-certs\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-c79hw\" (UID: \"bfae7061-0326-46fd-901b-0399080c2094\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-c79hw"
Oct 30 22:02:39.806967 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:39.799003    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/025aea26-9742-4272-9249-2e916c03d53d-istio-envoy\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-wg5rk\" (UID: \"025aea26-9742-4272-9249-2e916c03d53d\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-wg5rk"
Oct 30 22:02:39.806967 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:39.799034    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfae7061-0326-46fd-901b-0399080c2094-istiod-ca-cert\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-c79hw\" (UID: \"bfae7061-0326-46fd-901b-0399080c2094\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-c79hw"
Oct 30 22:02:39.806967 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:39.799061    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/025aea26-9742-4272-9249-2e916c03d53d-workload-certs\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-wg5rk\" (UID: \"025aea26-9742-4272-9249-2e916c03d53d\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-wg5rk"
Oct 30 22:02:39.806967 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:39.799146    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/025aea26-9742-4272-9249-2e916c03d53d-istio-podinfo\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-wg5rk\" (UID: \"025aea26-9742-4272-9249-2e916c03d53d\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-wg5rk"
Oct 30 22:02:39.806967 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:39.799181    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfae7061-0326-46fd-901b-0399080c2094-istio-data\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-c79hw\" (UID: \"bfae7061-0326-46fd-901b-0399080c2094\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-c79hw"
Oct 30 22:02:39.806967 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:39.799223    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfae7061-0326-46fd-901b-0399080c2094-istio-podinfo\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-c79hw\" (UID: \"bfae7061-0326-46fd-901b-0399080c2094\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-c79hw"
Oct 30 22:02:39.806967 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:39.799290    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/025aea26-9742-4272-9249-2e916c03d53d-workload-socket\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-wg5rk\" (UID: \"025aea26-9742-4272-9249-2e916c03d53d\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-wg5rk"
Oct 30 22:02:39.806967 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:39.799384    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/025aea26-9742-4272-9249-2e916c03d53d-istio-token\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-wg5rk\" (UID: \"025aea26-9742-4272-9249-2e916c03d53d\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-wg5rk"
Oct 30 22:02:39.806967 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:39.799411    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfae7061-0326-46fd-901b-0399080c2094-istio-envoy\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-c79hw\" (UID: \"bfae7061-0326-46fd-901b-0399080c2094\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-c79hw"
Oct 30 22:02:39.806967 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:39.799476    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/025aea26-9742-4272-9249-2e916c03d53d-credential-socket\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-wg5rk\" (UID: \"025aea26-9742-4272-9249-2e916c03d53d\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-wg5rk"
Oct 30 22:02:39.806967 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:39.799598    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfae7061-0326-46fd-901b-0399080c2094-istio-token\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-c79hw\" (UID: \"bfae7061-0326-46fd-901b-0399080c2094\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-c79hw"
Oct 30 22:02:39.806967 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:39.800615    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/025aea26-9742-4272-9249-2e916c03d53d-istio-data\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-wg5rk\" (UID: \"025aea26-9742-4272-9249-2e916c03d53d\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-wg5rk"
Oct 30 22:02:39.806967 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:39.800719    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/025aea26-9742-4272-9249-2e916c03d53d-workload-certs\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-wg5rk\" (UID: \"025aea26-9742-4272-9249-2e916c03d53d\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-wg5rk"
Oct 30 22:02:39.806967 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:39.801212    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/025aea26-9742-4272-9249-2e916c03d53d-istiod-ca-cert\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-wg5rk\" (UID: \"025aea26-9742-4272-9249-2e916c03d53d\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-wg5rk"
Oct 30 22:02:39.806967 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:39.801983    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/025aea26-9742-4272-9249-2e916c03d53d-workload-socket\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-wg5rk\" (UID: \"025aea26-9742-4272-9249-2e916c03d53d\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-wg5rk"
Oct 30 22:02:39.806967 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:39.802569    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/025aea26-9742-4272-9249-2e916c03d53d-credential-socket\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-wg5rk\" (UID: \"025aea26-9742-4272-9249-2e916c03d53d\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-wg5rk"
Oct 30 22:02:39.964555 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:39.924568    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45jmz\" (UniqueName: \"kubernetes.io/projected/025aea26-9742-4272-9249-2e916c03d53d-kube-api-access-45jmz\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-wg5rk\" (UID: \"025aea26-9742-4272-9249-2e916c03d53d\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-wg5rk"
Oct 30 22:02:39.964555 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:39.925571    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxlzx\" (UniqueName: \"kubernetes.io/projected/bfae7061-0326-46fd-901b-0399080c2094-kube-api-access-bxlzx\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-c79hw\" (UID: \"bfae7061-0326-46fd-901b-0399080c2094\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-c79hw"
Oct 30 22:02:39.964555 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:39.925624    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfae7061-0326-46fd-901b-0399080c2094-credential-socket\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-c79hw\" (UID: \"bfae7061-0326-46fd-901b-0399080c2094\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-c79hw"
Oct 30 22:02:39.964555 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:39.925687    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfae7061-0326-46fd-901b-0399080c2094-workload-certs\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-c79hw\" (UID: \"bfae7061-0326-46fd-901b-0399080c2094\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-c79hw"
Oct 30 22:02:39.964555 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:39.925733    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfae7061-0326-46fd-901b-0399080c2094-istiod-ca-cert\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-c79hw\" (UID: \"bfae7061-0326-46fd-901b-0399080c2094\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-c79hw"
Oct 30 22:02:39.964555 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:39.925827    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfae7061-0326-46fd-901b-0399080c2094-istio-data\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-c79hw\" (UID: \"bfae7061-0326-46fd-901b-0399080c2094\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-c79hw"
Oct 30 22:02:39.964555 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:39.925875    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfae7061-0326-46fd-901b-0399080c2094-istio-podinfo\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-c79hw\" (UID: \"bfae7061-0326-46fd-901b-0399080c2094\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-c79hw"
Oct 30 22:02:39.964555 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:39.925988    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfae7061-0326-46fd-901b-0399080c2094-istio-envoy\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-c79hw\" (UID: \"bfae7061-0326-46fd-901b-0399080c2094\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-c79hw"
Oct 30 22:02:39.964555 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:39.926094    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfae7061-0326-46fd-901b-0399080c2094-istio-token\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-c79hw\" (UID: \"bfae7061-0326-46fd-901b-0399080c2094\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-c79hw"
Oct 30 22:02:39.964555 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:39.926129    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfae7061-0326-46fd-901b-0399080c2094-workload-socket\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-c79hw\" (UID: \"bfae7061-0326-46fd-901b-0399080c2094\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-c79hw"
Oct 30 22:02:39.964555 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:39.926647    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfae7061-0326-46fd-901b-0399080c2094-workload-socket\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-c79hw\" (UID: \"bfae7061-0326-46fd-901b-0399080c2094\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-c79hw"
Oct 30 22:02:39.964555 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:39.927232    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfae7061-0326-46fd-901b-0399080c2094-credential-socket\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-c79hw\" (UID: \"bfae7061-0326-46fd-901b-0399080c2094\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-c79hw"
Oct 30 22:02:39.964555 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:39.927497    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfae7061-0326-46fd-901b-0399080c2094-workload-certs\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-c79hw\" (UID: \"bfae7061-0326-46fd-901b-0399080c2094\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-c79hw"
Oct 30 22:02:39.967568 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:39.966834    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfae7061-0326-46fd-901b-0399080c2094-istiod-ca-cert\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-c79hw\" (UID: \"bfae7061-0326-46fd-901b-0399080c2094\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-c79hw"
Oct 30 22:02:39.967568 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:39.967119    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfae7061-0326-46fd-901b-0399080c2094-istio-data\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-c79hw\" (UID: \"bfae7061-0326-46fd-901b-0399080c2094\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-c79hw"
Oct 30 22:02:40.006184 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:40.006145    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/025aea26-9742-4272-9249-2e916c03d53d-istio-envoy\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-wg5rk\" (UID: \"025aea26-9742-4272-9249-2e916c03d53d\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-wg5rk"
Oct 30 22:02:40.100635 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:40.094131    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfae7061-0326-46fd-901b-0399080c2094-istio-token\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-c79hw\" (UID: \"bfae7061-0326-46fd-901b-0399080c2094\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-c79hw"
Oct 30 22:02:40.100635 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:40.100395    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxlzx\" (UniqueName: \"kubernetes.io/projected/bfae7061-0326-46fd-901b-0399080c2094-kube-api-access-bxlzx\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-c79hw\" (UID: \"bfae7061-0326-46fd-901b-0399080c2094\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-c79hw"
Oct 30 22:02:40.103710 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:40.103658    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfae7061-0326-46fd-901b-0399080c2094-istio-podinfo\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-c79hw\" (UID: \"bfae7061-0326-46fd-901b-0399080c2094\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-c79hw"
Oct 30 22:02:40.103710 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:40.103701    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/025aea26-9742-4272-9249-2e916c03d53d-istio-token\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-wg5rk\" (UID: \"025aea26-9742-4272-9249-2e916c03d53d\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-wg5rk"
Oct 30 22:02:40.105391 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:40.105341    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfae7061-0326-46fd-901b-0399080c2094-istio-envoy\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-c79hw\" (UID: \"bfae7061-0326-46fd-901b-0399080c2094\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-c79hw"
Oct 30 22:02:40.106739 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:40.106699    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/025aea26-9742-4272-9249-2e916c03d53d-istio-podinfo\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-wg5rk\" (UID: \"025aea26-9742-4272-9249-2e916c03d53d\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-wg5rk"
Oct 30 22:02:40.140573 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:40.118549    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-wg5rk"
Oct 30 22:02:40.186227 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:40.141157    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-c79hw"
Oct 30 22:02:40.470999 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:40.470901    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-kdmlq" event={"ID":"1aec235a-1d74-4044-aa3d-bd87a6a2d86f","Type":"ContainerStarted","Data":"30a4ed5a6cb2e9a2845117356449fa90af9c06c206524fcf5bf3976b75992727"}
Oct 30 22:02:40.470999 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:40.470967    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-kdmlq" event={"ID":"1aec235a-1d74-4044-aa3d-bd87a6a2d86f","Type":"ContainerStarted","Data":"bf81e773ad5755d83b1aee2d238d9e855a27a44e799b994b6cdf66edfc3bfb54"}
Oct 30 22:02:40.585573 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:40.585492    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-kdmlq" podStartSLOduration=2.585476729 podStartE2EDuration="2.585476729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:02:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:02:40.583929891 +0000 UTC m=+1214.196699659" watchObservedRunningTime="2026-10-30 22:02:40.585476729 +0000 UTC m=+1214.198246486"
Oct 30 22:02:40.805003 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:40.804957    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-wg5rk"]
Oct 30 22:02:40.814843 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:40.809171    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-c79hw"]
Oct 30 22:02:40.817489 ip-10-0-130-60 kubenswrapper[5061]: W1030 22:02:40.817456    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod025aea26_9742_4272_9249_2e916c03d53d.slice/crio-6442fe84e95c817174a2792cdc26997d8658fdcd758283db8bdaeb2fb3baf61b WatchSource:0}: Error finding container 6442fe84e95c817174a2792cdc26997d8658fdcd758283db8bdaeb2fb3baf61b: Status 404 returned error can't find the container with id 6442fe84e95c817174a2792cdc26997d8658fdcd758283db8bdaeb2fb3baf61b
Oct 30 22:02:40.821196 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:40.821149    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:02:40.821317 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:40.821234    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:02:40.821317 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:40.821268    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:02:40.846876 ip-10-0-130-60 kubenswrapper[5061]: W1030 22:02:40.846840    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfae7061_0326_46fd_901b_0399080c2094.slice/crio-5e84dbcf17016e163621b5fd14f1b7f9bc87e63c98173240ad51ba66466e0bc5 WatchSource:0}: Error finding container 5e84dbcf17016e163621b5fd14f1b7f9bc87e63c98173240ad51ba66466e0bc5: Status 404 returned error can't find the container with id 5e84dbcf17016e163621b5fd14f1b7f9bc87e63c98173240ad51ba66466e0bc5
Oct 30 22:02:40.849379 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:40.849333    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:02:40.849462 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:40.849436    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:02:40.849509 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:40.849484    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:02:41.070670 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:41.070574    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-kdmlq"
Oct 30 22:02:41.072504 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:41.072395    5061 patch_prober.go:29] interesting pod/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-kdmlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 30 22:02:41.072504 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:41.072446    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-kdmlq" podUID="1aec235a-1d74-4044-aa3d-bd87a6a2d86f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 30 22:02:41.480648 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:41.480509    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-wg5rk" event={"ID":"025aea26-9742-4272-9249-2e916c03d53d","Type":"ContainerStarted","Data":"20f371dff9c18ed9ac092238deb6a2806e4a79f27afa197072a9fa0f4aae236c"}
Oct 30 22:02:41.480648 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:41.480610    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-wg5rk" event={"ID":"025aea26-9742-4272-9249-2e916c03d53d","Type":"ContainerStarted","Data":"6442fe84e95c817174a2792cdc26997d8658fdcd758283db8bdaeb2fb3baf61b"}
Oct 30 22:02:41.482398 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:41.482370    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-c79hw" event={"ID":"bfae7061-0326-46fd-901b-0399080c2094","Type":"ContainerStarted","Data":"888f926dda2182b4bd0ce60084acaa9343c93fc6be5e86dc479441e7cc6f50be"}
Oct 30 22:02:41.482562 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:41.482413    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-c79hw" event={"ID":"bfae7061-0326-46fd-901b-0399080c2094","Type":"ContainerStarted","Data":"5e84dbcf17016e163621b5fd14f1b7f9bc87e63c98173240ad51ba66466e0bc5"}
Oct 30 22:02:41.636549 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:41.633520    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-wg5rk" podStartSLOduration=2.633495409 podStartE2EDuration="2.633495409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:02:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:02:41.604007909 +0000 UTC m=+1215.216777705" watchObservedRunningTime="2026-10-30 22:02:41.633495409 +0000 UTC m=+1215.246265169"
Oct 30 22:02:42.086550 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:42.082728    5061 patch_prober.go:29] interesting pod/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-kdmlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 30 22:02:42.086550 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:42.082802    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-kdmlq" podUID="1aec235a-1d74-4044-aa3d-bd87a6a2d86f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 30 22:02:42.132742 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:42.132683    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-wg5rk"
Oct 30 22:02:42.178562 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:42.156909    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-c79hw"
Oct 30 22:02:42.178562 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:42.158268    5061 patch_prober.go:29] interesting pod/e2e-worker-w4-17d809-openshift-default-5b676976ff-wg5rk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Oct 30 22:02:42.178562 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:42.158330    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-wg5rk" podUID="025aea26-9742-4272-9249-2e916c03d53d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Oct 30 22:02:42.178562 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:42.159665    5061 patch_prober.go:29] interesting pod/e2e-worker-w4-17d809-openshift-default-5b676976ff-c79hw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Oct 30 22:02:42.178562 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:42.159712    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-c79hw" podUID="bfae7061-0326-46fd-901b-0399080c2094" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Oct 30 22:02:43.077552 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:43.076956    5061 patch_prober.go:29] interesting pod/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-kdmlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 30 22:02:43.077552 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:43.077037    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-kdmlq" podUID="1aec235a-1d74-4044-aa3d-bd87a6a2d86f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 30 22:02:43.132549 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:43.128858    5061 patch_prober.go:29] interesting pod/e2e-worker-w4-17d809-openshift-default-5b676976ff-wg5rk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Oct 30 22:02:43.132549 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:43.128936    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-wg5rk" podUID="025aea26-9742-4272-9249-2e916c03d53d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Oct 30 22:02:43.154687 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:43.154644    5061 patch_prober.go:29] interesting pod/e2e-worker-w4-17d809-openshift-default-5b676976ff-c79hw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Oct 30 22:02:43.154885 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:43.154711    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-c79hw" podUID="bfae7061-0326-46fd-901b-0399080c2094" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Oct 30 22:02:44.071552 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:44.070545    5061 patch_prober.go:29] interesting pod/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-kdmlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 30 22:02:44.071552 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:44.070624    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-kdmlq" podUID="1aec235a-1d74-4044-aa3d-bd87a6a2d86f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 30 22:02:44.145559 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:44.141786    5061 patch_prober.go:29] interesting pod/e2e-worker-w4-17d809-openshift-default-5b676976ff-c79hw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Oct 30 22:02:44.145559 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:44.141850    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-c79hw" podUID="bfae7061-0326-46fd-901b-0399080c2094" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Oct 30 22:02:44.145559 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:44.143976    5061 patch_prober.go:29] interesting pod/e2e-worker-w4-17d809-openshift-default-5b676976ff-wg5rk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Oct 30 22:02:44.145559 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:44.144021    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-wg5rk" podUID="025aea26-9742-4272-9249-2e916c03d53d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Oct 30 22:02:45.078975 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:45.078939    5061 patch_prober.go:29] interesting pod/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-kdmlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 30 22:02:45.079301 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:45.079253    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-kdmlq" podUID="1aec235a-1d74-4044-aa3d-bd87a6a2d86f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 30 22:02:45.149556 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:45.129941    5061 patch_prober.go:29] interesting pod/e2e-worker-w4-17d809-openshift-default-5b676976ff-wg5rk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Oct 30 22:02:45.149556 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:45.130004    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-wg5rk" podUID="025aea26-9742-4272-9249-2e916c03d53d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Oct 30 22:02:45.191893 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:45.191845    5061 patch_prober.go:29] interesting pod/e2e-worker-w4-17d809-openshift-default-5b676976ff-c79hw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Oct 30 22:02:45.192173 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:45.192144    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-c79hw" podUID="bfae7061-0326-46fd-901b-0399080c2094" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Oct 30 22:02:46.074549 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:46.073350    5061 patch_prober.go:29] interesting pod/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-kdmlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 30 22:02:46.074549 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:46.073413    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-kdmlq" podUID="1aec235a-1d74-4044-aa3d-bd87a6a2d86f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 30 22:02:46.129466 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:46.129429    5061 patch_prober.go:29] interesting pod/e2e-worker-w4-17d809-openshift-default-5b676976ff-wg5rk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Oct 30 22:02:46.129763 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:46.129732    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-wg5rk" podUID="025aea26-9742-4272-9249-2e916c03d53d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Oct 30 22:02:46.163694 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:46.163656    5061 patch_prober.go:29] interesting pod/e2e-worker-w4-17d809-openshift-default-5b676976ff-c79hw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Oct 30 22:02:46.165265 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:46.165230    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-c79hw" podUID="bfae7061-0326-46fd-901b-0399080c2094" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Oct 30 22:02:47.070509 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:47.070458    5061 patch_prober.go:29] interesting pod/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-kdmlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 30 22:02:47.070730 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:47.070521    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-kdmlq" podUID="1aec235a-1d74-4044-aa3d-bd87a6a2d86f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 30 22:02:47.132552 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:47.132255    5061 patch_prober.go:29] interesting pod/e2e-worker-w4-17d809-openshift-default-5b676976ff-wg5rk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Oct 30 22:02:47.132552 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:47.132311    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-wg5rk" podUID="025aea26-9742-4272-9249-2e916c03d53d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Oct 30 22:02:47.145552 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:47.144918    5061 patch_prober.go:29] interesting pod/e2e-worker-w4-17d809-openshift-default-5b676976ff-c79hw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Oct 30 22:02:47.145552 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:47.144973    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-c79hw" podUID="bfae7061-0326-46fd-901b-0399080c2094" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Oct 30 22:02:48.125183 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:48.125143    5061 patch_prober.go:29] interesting pod/e2e-worker-w4-17d809-openshift-default-5b676976ff-wg5rk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Oct 30 22:02:48.125775 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:48.125213    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-wg5rk" podUID="025aea26-9742-4272-9249-2e916c03d53d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Oct 30 22:02:48.142180 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:48.142145    5061 patch_prober.go:29] interesting pod/e2e-worker-w4-17d809-openshift-default-5b676976ff-c79hw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Oct 30 22:02:48.142344 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:48.142215    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-c79hw" podUID="bfae7061-0326-46fd-901b-0399080c2094" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Oct 30 22:02:48.160953 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:48.160920    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-kdmlq"
Oct 30 22:02:48.161993 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:48.161966    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-kdmlq"
Oct 30 22:02:48.163966 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:48.163946    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-kdmlq"
Oct 30 22:02:48.290635 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:48.289486    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-c79hw" podStartSLOduration=9.289468588 podStartE2EDuration="9.289468588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:02:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:02:41.742708526 +0000 UTC m=+1215.355478297" watchObservedRunningTime="2026-10-30 22:02:48.289468588 +0000 UTC m=+1221.902238346"
Oct 30 22:02:49.129466 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:49.129433    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-wg5rk"
Oct 30 22:02:49.129935 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:49.129846    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-wg5rk"
Oct 30 22:02:49.130778 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:49.130747    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-wg5rk"
Oct 30 22:02:49.147148 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:49.147119    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-c79hw"
Oct 30 22:02:49.147407 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:49.147386    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-c79hw"
Oct 30 22:02:49.148231 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:49.148212    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-c79hw"
Oct 30 22:02:54.053521 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:54.053486    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03e931f4-gbsdp"]
Oct 30 22:02:54.100650 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:54.100617    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03e931f4-gbsdp"]
Oct 30 22:02:54.530238 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:54.530153    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ebb4214e-0beb-474e-aa33-d9ebf9229826" path="/var/lib/kubelet/pods/ebb4214e-0beb-474e-aa33-d9ebf9229826/volumes"
Oct 30 22:02:54.668491 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:02:54.668455    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 22:02:54.929956 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:54.929917    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-n4bnf"]
Oct 30 22:02:54.934760 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:54.934738    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-n4bnf"
Oct 30 22:02:54.996562 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:54.996501    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-n4bnf"]
Oct 30 22:02:55.016240 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:55.016204    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/799953d1-1dc8-421e-9e98-d2696d399ac2-istiod-ca-cert\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-n4bnf\" (UID: \"799953d1-1dc8-421e-9e98-d2696d399ac2\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-n4bnf"
Oct 30 22:02:55.016240 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:55.016242    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/799953d1-1dc8-421e-9e98-d2696d399ac2-credential-socket\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-n4bnf\" (UID: \"799953d1-1dc8-421e-9e98-d2696d399ac2\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-n4bnf"
Oct 30 22:02:55.016492 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:55.016267    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/799953d1-1dc8-421e-9e98-d2696d399ac2-workload-certs\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-n4bnf\" (UID: \"799953d1-1dc8-421e-9e98-d2696d399ac2\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-n4bnf"
Oct 30 22:02:55.016492 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:55.016392    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjm2l\" (UniqueName: \"kubernetes.io/projected/799953d1-1dc8-421e-9e98-d2696d399ac2-kube-api-access-gjm2l\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-n4bnf\" (UID: \"799953d1-1dc8-421e-9e98-d2696d399ac2\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-n4bnf"
Oct 30 22:02:55.016492 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:55.016423    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/799953d1-1dc8-421e-9e98-d2696d399ac2-istio-data\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-n4bnf\" (UID: \"799953d1-1dc8-421e-9e98-d2696d399ac2\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-n4bnf"
Oct 30 22:02:55.016492 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:55.016448    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/799953d1-1dc8-421e-9e98-d2696d399ac2-istio-podinfo\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-n4bnf\" (UID: \"799953d1-1dc8-421e-9e98-d2696d399ac2\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-n4bnf"
Oct 30 22:02:55.016739 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:55.016553    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/799953d1-1dc8-421e-9e98-d2696d399ac2-istio-envoy\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-n4bnf\" (UID: \"799953d1-1dc8-421e-9e98-d2696d399ac2\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-n4bnf"
Oct 30 22:02:55.016739 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:55.016675    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/799953d1-1dc8-421e-9e98-d2696d399ac2-workload-socket\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-n4bnf\" (UID: \"799953d1-1dc8-421e-9e98-d2696d399ac2\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-n4bnf"
Oct 30 22:02:55.016739 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:55.016712    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/799953d1-1dc8-421e-9e98-d2696d399ac2-istio-token\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-n4bnf\" (UID: \"799953d1-1dc8-421e-9e98-d2696d399ac2\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-n4bnf"
Oct 30 22:02:55.118346 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:55.118307    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/799953d1-1dc8-421e-9e98-d2696d399ac2-istiod-ca-cert\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-n4bnf\" (UID: \"799953d1-1dc8-421e-9e98-d2696d399ac2\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-n4bnf"
Oct 30 22:02:55.118346 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:55.118352    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/799953d1-1dc8-421e-9e98-d2696d399ac2-credential-socket\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-n4bnf\" (UID: \"799953d1-1dc8-421e-9e98-d2696d399ac2\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-n4bnf"
Oct 30 22:02:55.118918 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:55.118385    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/799953d1-1dc8-421e-9e98-d2696d399ac2-workload-certs\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-n4bnf\" (UID: \"799953d1-1dc8-421e-9e98-d2696d399ac2\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-n4bnf"
Oct 30 22:02:55.118918 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:55.118432    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjm2l\" (UniqueName: \"kubernetes.io/projected/799953d1-1dc8-421e-9e98-d2696d399ac2-kube-api-access-gjm2l\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-n4bnf\" (UID: \"799953d1-1dc8-421e-9e98-d2696d399ac2\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-n4bnf"
Oct 30 22:02:55.118918 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:55.118476    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/799953d1-1dc8-421e-9e98-d2696d399ac2-istio-data\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-n4bnf\" (UID: \"799953d1-1dc8-421e-9e98-d2696d399ac2\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-n4bnf"
Oct 30 22:02:55.118918 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:55.118507    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/799953d1-1dc8-421e-9e98-d2696d399ac2-istio-podinfo\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-n4bnf\" (UID: \"799953d1-1dc8-421e-9e98-d2696d399ac2\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-n4bnf"
Oct 30 22:02:55.118918 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:55.118582    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/799953d1-1dc8-421e-9e98-d2696d399ac2-istio-envoy\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-n4bnf\" (UID: \"799953d1-1dc8-421e-9e98-d2696d399ac2\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-n4bnf"
Oct 30 22:02:55.118918 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:55.118655    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/799953d1-1dc8-421e-9e98-d2696d399ac2-workload-socket\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-n4bnf\" (UID: \"799953d1-1dc8-421e-9e98-d2696d399ac2\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-n4bnf"
Oct 30 22:02:55.118918 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:55.118679    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/799953d1-1dc8-421e-9e98-d2696d399ac2-istio-token\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-n4bnf\" (UID: \"799953d1-1dc8-421e-9e98-d2696d399ac2\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-n4bnf"
Oct 30 22:02:55.119279 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:55.119031    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/799953d1-1dc8-421e-9e98-d2696d399ac2-credential-socket\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-n4bnf\" (UID: \"799953d1-1dc8-421e-9e98-d2696d399ac2\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-n4bnf"
Oct 30 22:02:55.119279 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:55.119041    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/799953d1-1dc8-421e-9e98-d2696d399ac2-workload-certs\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-n4bnf\" (UID: \"799953d1-1dc8-421e-9e98-d2696d399ac2\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-n4bnf"
Oct 30 22:02:55.119279 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:55.119265    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/799953d1-1dc8-421e-9e98-d2696d399ac2-istio-data\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-n4bnf\" (UID: \"799953d1-1dc8-421e-9e98-d2696d399ac2\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-n4bnf"
Oct 30 22:02:55.119440 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:55.119301    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/799953d1-1dc8-421e-9e98-d2696d399ac2-istiod-ca-cert\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-n4bnf\" (UID: \"799953d1-1dc8-421e-9e98-d2696d399ac2\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-n4bnf"
Oct 30 22:02:55.119440 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:55.119317    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/799953d1-1dc8-421e-9e98-d2696d399ac2-workload-socket\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-n4bnf\" (UID: \"799953d1-1dc8-421e-9e98-d2696d399ac2\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-n4bnf"
Oct 30 22:02:55.121204 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:55.121182    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/799953d1-1dc8-421e-9e98-d2696d399ac2-istio-envoy\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-n4bnf\" (UID: \"799953d1-1dc8-421e-9e98-d2696d399ac2\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-n4bnf"
Oct 30 22:02:55.121313 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:55.121287    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/799953d1-1dc8-421e-9e98-d2696d399ac2-istio-podinfo\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-n4bnf\" (UID: \"799953d1-1dc8-421e-9e98-d2696d399ac2\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-n4bnf"
Oct 30 22:02:55.173807 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:55.173772    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/799953d1-1dc8-421e-9e98-d2696d399ac2-istio-token\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-n4bnf\" (UID: \"799953d1-1dc8-421e-9e98-d2696d399ac2\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-n4bnf"
Oct 30 22:02:55.173970 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:55.173928    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjm2l\" (UniqueName: \"kubernetes.io/projected/799953d1-1dc8-421e-9e98-d2696d399ac2-kube-api-access-gjm2l\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-n4bnf\" (UID: \"799953d1-1dc8-421e-9e98-d2696d399ac2\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-n4bnf"
Oct 30 22:02:55.249318 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:55.249236    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-n4bnf"
Oct 30 22:02:55.681001 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:55.680968    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-n4bnf"]
Oct 30 22:02:55.722644 ip-10-0-130-60 kubenswrapper[5061]: W1030 22:02:55.722605    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod799953d1_1dc8_421e_9e98_d2696d399ac2.slice/crio-7995f901a3e0e9925c44d43afa72388bb6d450c87cd051cfaf794141c9cfae04 WatchSource:0}: Error finding container 7995f901a3e0e9925c44d43afa72388bb6d450c87cd051cfaf794141c9cfae04: Status 404 returned error can't find the container with id 7995f901a3e0e9925c44d43afa72388bb6d450c87cd051cfaf794141c9cfae04
Oct 30 22:02:55.725250 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:55.725217    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:02:55.725331 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:55.725287    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:02:55.725331 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:55.725316    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:02:56.647815 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:56.647775    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-n4bnf" event={"ID":"799953d1-1dc8-421e-9e98-d2696d399ac2","Type":"ContainerStarted","Data":"8eddbddec2bf4fa136c7343e98b86ac5eb26c4de459abf34ac50a87be0e862a6"}
Oct 30 22:02:56.648265 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:56.647832    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-n4bnf" event={"ID":"799953d1-1dc8-421e-9e98-d2696d399ac2","Type":"ContainerStarted","Data":"7995f901a3e0e9925c44d43afa72388bb6d450c87cd051cfaf794141c9cfae04"}
Oct 30 22:02:56.881224 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:56.881173    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-n4bnf" podStartSLOduration=2.881159373 podStartE2EDuration="2.881159373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:02:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:02:56.83306143 +0000 UTC m=+1230.445831222" watchObservedRunningTime="2026-10-30 22:02:56.881159373 +0000 UTC m=+1230.493929130"
Oct 30 22:02:57.250195 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:57.250163    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-n4bnf"
Oct 30 22:02:57.251728 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:57.251701    5061 patch_prober.go:29] interesting pod/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-n4bnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Oct 30 22:02:57.251859 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:57.251751    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-n4bnf" podUID="799953d1-1dc8-421e-9e98-d2696d399ac2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Oct 30 22:02:58.250282 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:58.250246    5061 patch_prober.go:29] interesting pod/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-n4bnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Oct 30 22:02:58.250800 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:58.250302    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-n4bnf" podUID="799953d1-1dc8-421e-9e98-d2696d399ac2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Oct 30 22:02:59.250407 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:59.250375    5061 patch_prober.go:29] interesting pod/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-n4bnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Oct 30 22:02:59.250853 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:02:59.250424    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-n4bnf" podUID="799953d1-1dc8-421e-9e98-d2696d399ac2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Oct 30 22:03:00.249735 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:00.249696    5061 patch_prober.go:29] interesting pod/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-n4bnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Oct 30 22:03:00.249935 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:00.249777    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-n4bnf" podUID="799953d1-1dc8-421e-9e98-d2696d399ac2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Oct 30 22:03:01.250323 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:01.250286    5061 patch_prober.go:29] interesting pod/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-n4bnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Oct 30 22:03:01.250747 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:01.250350    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-n4bnf" podUID="799953d1-1dc8-421e-9e98-d2696d399ac2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Oct 30 22:03:02.250653 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:02.250614    5061 patch_prober.go:29] interesting pod/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-n4bnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Oct 30 22:03:02.251133 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:02.250683    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-n4bnf" podUID="799953d1-1dc8-421e-9e98-d2696d399ac2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Oct 30 22:03:03.255827 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:03.255792    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-n4bnf"
Oct 30 22:03:03.256444 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:03.256415    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-n4bnf"
Oct 30 22:03:03.257220 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:03.257157    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-n4bnf"
Oct 30 22:03:09.049296 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:09.049258    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-343fe207-chkvn"]
Oct 30 22:03:09.095970 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:09.095937    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-343fe207-chkvn"]
Oct 30 22:03:10.560553 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:10.556756    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="113abc93-9162-4924-a232-29bceb7ef71d" path="/var/lib/kubelet/pods/113abc93-9162-4924-a232-29bceb7ef71d/volumes"
Oct 30 22:03:24.676025 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:03:24.675980    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 22:03:26.783743 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:26.783712    5061 scope.go:122] "RemoveContainer" containerID="2eec9d5eb1e1b23e1150255a8ef82fced17e0b7952ea0c79087871b411246476"
Oct 30 22:03:29.094785 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:29.094751    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-llrwj-vhl5p"]
Oct 30 22:03:29.100984 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:29.100964    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-llrwj-vhl5p"
Oct 30 22:03:29.119459 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:29.119431    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/661f1c2f-0035-455b-8dfe-21d798d35552-archives-path\") pod \"periodic-gathering-llrwj-vhl5p\" (UID: \"661f1c2f-0035-455b-8dfe-21d798d35552\") " pod="openshift-insights/periodic-gathering-llrwj-vhl5p"
Oct 30 22:03:29.119629 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:29.119612    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5785c\" (UniqueName: \"kubernetes.io/projected/661f1c2f-0035-455b-8dfe-21d798d35552-kube-api-access-5785c\") pod \"periodic-gathering-llrwj-vhl5p\" (UID: \"661f1c2f-0035-455b-8dfe-21d798d35552\") " pod="openshift-insights/periodic-gathering-llrwj-vhl5p"
Oct 30 22:03:29.119763 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:29.119746    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/661f1c2f-0035-455b-8dfe-21d798d35552-service-ca-bundle\") pod \"periodic-gathering-llrwj-vhl5p\" (UID: \"661f1c2f-0035-455b-8dfe-21d798d35552\") " pod="openshift-insights/periodic-gathering-llrwj-vhl5p"
Oct 30 22:03:29.119832 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:29.119805    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/661f1c2f-0035-455b-8dfe-21d798d35552-trusted-ca-bundle\") pod \"periodic-gathering-llrwj-vhl5p\" (UID: \"661f1c2f-0035-455b-8dfe-21d798d35552\") " pod="openshift-insights/periodic-gathering-llrwj-vhl5p"
Oct 30 22:03:29.169937 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:29.169910    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-llrwj-vhl5p"]
Oct 30 22:03:29.220766 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:29.220733    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5785c\" (UniqueName: \"kubernetes.io/projected/661f1c2f-0035-455b-8dfe-21d798d35552-kube-api-access-5785c\") pod \"periodic-gathering-llrwj-vhl5p\" (UID: \"661f1c2f-0035-455b-8dfe-21d798d35552\") " pod="openshift-insights/periodic-gathering-llrwj-vhl5p"
Oct 30 22:03:29.221052 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:29.220882    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/661f1c2f-0035-455b-8dfe-21d798d35552-service-ca-bundle\") pod \"periodic-gathering-llrwj-vhl5p\" (UID: \"661f1c2f-0035-455b-8dfe-21d798d35552\") " pod="openshift-insights/periodic-gathering-llrwj-vhl5p"
Oct 30 22:03:29.221052 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:29.220951    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/661f1c2f-0035-455b-8dfe-21d798d35552-trusted-ca-bundle\") pod \"periodic-gathering-llrwj-vhl5p\" (UID: \"661f1c2f-0035-455b-8dfe-21d798d35552\") " pod="openshift-insights/periodic-gathering-llrwj-vhl5p"
Oct 30 22:03:29.221209 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:29.221054    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/661f1c2f-0035-455b-8dfe-21d798d35552-archives-path\") pod \"periodic-gathering-llrwj-vhl5p\" (UID: \"661f1c2f-0035-455b-8dfe-21d798d35552\") " pod="openshift-insights/periodic-gathering-llrwj-vhl5p"
Oct 30 22:03:29.221478 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:29.221458    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/661f1c2f-0035-455b-8dfe-21d798d35552-archives-path\") pod \"periodic-gathering-llrwj-vhl5p\" (UID: \"661f1c2f-0035-455b-8dfe-21d798d35552\") " pod="openshift-insights/periodic-gathering-llrwj-vhl5p"
Oct 30 22:03:29.221630 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:29.221611    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/661f1c2f-0035-455b-8dfe-21d798d35552-service-ca-bundle\") pod \"periodic-gathering-llrwj-vhl5p\" (UID: \"661f1c2f-0035-455b-8dfe-21d798d35552\") " pod="openshift-insights/periodic-gathering-llrwj-vhl5p"
Oct 30 22:03:29.221954 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:29.221930    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/661f1c2f-0035-455b-8dfe-21d798d35552-trusted-ca-bundle\") pod \"periodic-gathering-llrwj-vhl5p\" (UID: \"661f1c2f-0035-455b-8dfe-21d798d35552\") " pod="openshift-insights/periodic-gathering-llrwj-vhl5p"
Oct 30 22:03:29.275360 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:29.275332    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5785c\" (UniqueName: \"kubernetes.io/projected/661f1c2f-0035-455b-8dfe-21d798d35552-kube-api-access-5785c\") pod \"periodic-gathering-llrwj-vhl5p\" (UID: \"661f1c2f-0035-455b-8dfe-21d798d35552\") " pod="openshift-insights/periodic-gathering-llrwj-vhl5p"
Oct 30 22:03:29.414246 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:29.414161    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-llrwj-vhl5p"
Oct 30 22:03:29.781981 ip-10-0-130-60 kubenswrapper[5061]: W1030 22:03:29.781950    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod661f1c2f_0035_455b_8dfe_21d798d35552.slice/crio-c8f0c6ee2e808b0b42f614df38bb609f6e000c2915ced8813ad735e730cc5872 WatchSource:0}: Error finding container c8f0c6ee2e808b0b42f614df38bb609f6e000c2915ced8813ad735e730cc5872: Status 404 returned error can't find the container with id c8f0c6ee2e808b0b42f614df38bb609f6e000c2915ced8813ad735e730cc5872
Oct 30 22:03:29.782647 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:29.782617    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-llrwj-vhl5p"]
Oct 30 22:03:29.958628 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:29.958585    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-llrwj-vhl5p" event={"ID":"661f1c2f-0035-455b-8dfe-21d798d35552","Type":"ContainerStarted","Data":"c3dd2a5a9ef8e07e73b7bd91cd7168349da30625a3806f69b8d79e575826e91a"}
Oct 30 22:03:29.958798 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:29.958643    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-llrwj-vhl5p" event={"ID":"661f1c2f-0035-455b-8dfe-21d798d35552","Type":"ContainerStarted","Data":"c8f0c6ee2e808b0b42f614df38bb609f6e000c2915ced8813ad735e730cc5872"}
Oct 30 22:03:30.071061 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:30.071012    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-llrwj-vhl5p" podStartSLOduration=1.070999723 podStartE2EDuration="1.070999723s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:03:30.067628689 +0000 UTC m=+1263.680398466" watchObservedRunningTime="2026-10-30 22:03:30.070999723 +0000 UTC m=+1263.683769481"
Oct 30 22:03:34.096554 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:34.091639    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-14af5c/premium-w6-14af5c-kserve-6549647f4-f4gwg" podUID="9b42ee01-c60e-462c-a0e0-b04a52014db6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:03:34.112561 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:34.110624    5061 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-5be0d1/distinct-w2-5be0d1-kserve-6b85698c8-sxn9k" podUID="7a2dbfc5-908f-41d1-baff-074c56d2cf16" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:03:34.159553 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:34.151196    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-17d809/distinct-w4-17d809-kserve-6b8f5c48cd-lhrwp" podUID="ef5e8691-42e1-471b-ba7d-90881028eca7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:03:34.182557 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:34.160634    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-blrrb" podUID="cb4b1ed5-2192-4877-8be7-ca82af831af5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": context deadline exceeded"
Oct 30 22:03:34.242551 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:34.241844    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5be0d1/distinct-w2-5be0d1-kserve-6b85698c8-sxn9k" podUID="7a2dbfc5-908f-41d1-baff-074c56d2cf16" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:03:34.293556 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:34.292627    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5be0d1/premium-w2-5be0d1-kserve-759b6fb995-c6gp4" podUID="933f2470-68de-4768-a05b-6c454f16f5f2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:03:34.345552 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:34.333629    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/distinct-2-w3-f776cb-kserve-7b4d4cc954-d88pf" podUID="5b184062-7fac-43d4-ba5f-21567df85da3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:03:34.345552 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:34.334067    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-17d809/premium-w4-17d809-kserve-7c9bdbbcbc-5mtmm" podUID="1e0eca81-7db1-482b-8dd6-5b28fb89e010" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Oct 30 22:03:34.407615 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:34.407075    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2d1f91/simulator-w0-2d1f91-kserve-7699765c74-jwtkb" podUID="44b7d6f6-32d8-4ef4-95bc-eef604fa6b5b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:03:34.636565 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:34.626162    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a07620/simulator-w1-a07620-kserve-55b44674f4-4z9lk" podUID="efe7c063-fb0e-4679-8a7c-13f2a0abaa74" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:03:34.670940 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:34.640002    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-14af5c/distinct-2-w6-14af5c-kserve-58d554bdb6-r5r6j" podUID="990f4e94-708e-45f3-b2ad-01cd50b3257b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded"
Oct 30 22:03:34.683757 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:34.683500    5061 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-17d809/distinct-w4-17d809-kserve-6b8f5c48cd-lhrwp" podUID="ef5e8691-42e1-471b-ba7d-90881028eca7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/health\": context deadline exceeded"
Oct 30 22:03:34.759640 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:34.757349    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-14af5c/distinct-w6-14af5c-kserve-7c7cb9d69c-hljl9" podUID="1f8d9486-fb2f-491a-a8d7-c23c21bd116a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:03:34.862043 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:34.862000    5061 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-f776cb/distinct-2-w3-f776cb-kserve-7b4d4cc954-d88pf" podUID="5b184062-7fac-43d4-ba5f-21567df85da3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:03:34.870558 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:34.867192    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2d1f91/premium-w0-2d1f91-kserve-6965b558cf-59lgw" podUID="fb84233d-25e0-48bc-8899-c3a3cb3cfeb9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 30 22:03:34.888523 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:34.888471    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-17d809/simulator-w4-17d809-kserve-78d67b6f4-7hsvb" podUID="e7337cf8-4f59-4afb-9b50-6b6c9fa4a773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:03:34.915653 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:34.895847    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-14af5c/simulator-w6-14af5c-kserve-85b8bd96d7-wlcb9" podUID="39766c7d-5f1d-42db-8920-c0321cd12c99" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:03:35.111582 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:35.094615    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-14af5c/premium-w6-14af5c-kserve-6549647f4-f4gwg" podUID="9b42ee01-c60e-462c-a0e0-b04a52014db6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:03:35.111582 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:35.095047    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-14af5c/premium-w6-14af5c-kserve-6549647f4-f4gwg"
Oct 30 22:03:35.117124 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:35.117080    5061 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-2d1f91/premium-w0-2d1f91-kserve-6965b558cf-59lgw" podUID="fb84233d-25e0-48bc-8899-c3a3cb3cfeb9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:03:35.164438 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:35.164395    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-blrrb" podUID="cb4b1ed5-2192-4877-8be7-ca82af831af5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:03:35.165049 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:35.165025    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-blrrb"
Oct 30 22:03:35.211552 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:35.166138    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-17d809/distinct-w4-17d809-kserve-6b8f5c48cd-lhrwp" podUID="ef5e8691-42e1-471b-ba7d-90881028eca7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:03:35.212213 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:35.212183    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-17d809/distinct-w4-17d809-kserve-6b8f5c48cd-lhrwp"
Oct 30 22:03:35.267660 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:35.267611    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5be0d1/distinct-w2-5be0d1-kserve-6b85698c8-sxn9k" podUID="7a2dbfc5-908f-41d1-baff-074c56d2cf16" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:03:35.268298 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:35.268273    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-5be0d1/distinct-w2-5be0d1-kserve-6b85698c8-sxn9k"
Oct 30 22:03:35.483612 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:35.393879    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/distinct-2-w3-f776cb-kserve-7b4d4cc954-d88pf" podUID="5b184062-7fac-43d4-ba5f-21567df85da3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded"
Oct 30 22:03:35.483612 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:35.403619    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-17d809/premium-w4-17d809-kserve-7c9bdbbcbc-5mtmm" podUID="1e0eca81-7db1-482b-8dd6-5b28fb89e010" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Oct 30 22:03:35.483612 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:35.404053    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f776cb/distinct-2-w3-f776cb-kserve-7b4d4cc954-d88pf"
Oct 30 22:03:35.483612 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:35.404117    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-17d809/premium-w4-17d809-kserve-7c9bdbbcbc-5mtmm"
Oct 30 22:03:35.519585 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:35.493426    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5be0d1/premium-w2-5be0d1-kserve-759b6fb995-c6gp4" podUID="933f2470-68de-4768-a05b-6c454f16f5f2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded"
Oct 30 22:03:35.519585 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:35.493826    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2d1f91/simulator-w0-2d1f91-kserve-7699765c74-jwtkb" podUID="44b7d6f6-32d8-4ef4-95bc-eef604fa6b5b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 22:03:35.519585 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:35.494152    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-5be0d1/premium-w2-5be0d1-kserve-759b6fb995-c6gp4"
Oct 30 22:03:35.534553 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:35.522581    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2d1f91/simulator-w0-2d1f91-kserve-7699765c74-jwtkb"
Oct 30 22:03:35.646717 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:35.646669    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a07620/simulator-w1-a07620-kserve-55b44674f4-4z9lk" podUID="efe7c063-fb0e-4679-8a7c-13f2a0abaa74" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Oct 30 22:03:35.652802 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:35.652773    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a07620/simulator-w1-a07620-kserve-55b44674f4-4z9lk"
Oct 30 22:03:35.734707 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:35.734600    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-14af5c/distinct-2-w6-14af5c-kserve-58d554bdb6-r5r6j" podUID="990f4e94-708e-45f3-b2ad-01cd50b3257b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:03:35.738551 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:35.735223    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-14af5c/distinct-2-w6-14af5c-kserve-58d554bdb6-r5r6j"
Oct 30 22:03:35.859640 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:35.828196    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-14af5c/distinct-w6-14af5c-kserve-7c7cb9d69c-hljl9" podUID="1f8d9486-fb2f-491a-a8d7-c23c21bd116a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded"
Oct 30 22:03:35.859640 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:35.828812    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-14af5c/distinct-w6-14af5c-kserve-7c7cb9d69c-hljl9"
Oct 30 22:03:36.060176 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:36.031436    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-14af5c/simulator-w6-14af5c-kserve-85b8bd96d7-wlcb9" podUID="39766c7d-5f1d-42db-8920-c0321cd12c99" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:03:36.060176 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:36.032337    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-17d809/simulator-w4-17d809-kserve-78d67b6f4-7hsvb" podUID="e7337cf8-4f59-4afb-9b50-6b6c9fa4a773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:03:36.060176 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:36.032623    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-14af5c/simulator-w6-14af5c-kserve-85b8bd96d7-wlcb9"
Oct 30 22:03:36.060176 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:36.032691    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-17d809/simulator-w4-17d809-kserve-78d67b6f4-7hsvb"
Oct 30 22:03:36.102704 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:36.091628    5061 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-5be0d1/premium-w2-5be0d1-kserve-759b6fb995-c6gp4" podUID="933f2470-68de-4768-a05b-6c454f16f5f2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:03:36.140622 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:36.106509    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-14af5c/premium-w6-14af5c-kserve-6549647f4-f4gwg" podUID="9b42ee01-c60e-462c-a0e0-b04a52014db6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:03:36.141122 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:36.140800    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2d1f91/premium-w0-2d1f91-kserve-6965b558cf-59lgw" podUID="fb84233d-25e0-48bc-8899-c3a3cb3cfeb9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 30 22:03:36.143003 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:36.142983    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2d1f91/premium-w0-2d1f91-kserve-6965b558cf-59lgw"
Oct 30 22:03:36.171558 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:36.170337    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-blrrb" podUID="cb4b1ed5-2192-4877-8be7-ca82af831af5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:03:36.228109 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:36.228065    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-17d809/distinct-w4-17d809-kserve-6b8f5c48cd-lhrwp" podUID="ef5e8691-42e1-471b-ba7d-90881028eca7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:03:36.287692 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:36.287641    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5be0d1/distinct-w2-5be0d1-kserve-6b85698c8-sxn9k" podUID="7a2dbfc5-908f-41d1-baff-074c56d2cf16" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:03:36.425607 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:36.400220    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/distinct-2-w3-f776cb-kserve-7b4d4cc954-d88pf" podUID="5b184062-7fac-43d4-ba5f-21567df85da3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:03:36.439676 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:36.439640    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-17d809/premium-w4-17d809-kserve-7c9bdbbcbc-5mtmm" podUID="1e0eca81-7db1-482b-8dd6-5b28fb89e010" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:03:36.516552 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:36.498635    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2d1f91/simulator-w0-2d1f91-kserve-7699765c74-jwtkb" podUID="44b7d6f6-32d8-4ef4-95bc-eef604fa6b5b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 22:03:36.516836 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:36.516801    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5be0d1/premium-w2-5be0d1-kserve-759b6fb995-c6gp4" podUID="933f2470-68de-4768-a05b-6c454f16f5f2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded"
Oct 30 22:03:36.654551 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:36.654362    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a07620/simulator-w1-a07620-kserve-55b44674f4-4z9lk" podUID="efe7c063-fb0e-4679-8a7c-13f2a0abaa74" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Oct 30 22:03:36.747913 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:36.747820    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-14af5c/distinct-2-w6-14af5c-kserve-58d554bdb6-r5r6j" podUID="990f4e94-708e-45f3-b2ad-01cd50b3257b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded"
Oct 30 22:03:36.843553 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:36.829822    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-14af5c/distinct-w6-14af5c-kserve-7c7cb9d69c-hljl9" podUID="1f8d9486-fb2f-491a-a8d7-c23c21bd116a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded"
Oct 30 22:03:36.927587 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:36.927379    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f776cb/distinct-2-w3-f776cb-kserve-7b4d4cc954-d88pf"
Oct 30 22:03:37.043231 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:37.043143    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-17d809/simulator-w4-17d809-kserve-78d67b6f4-7hsvb" podUID="e7337cf8-4f59-4afb-9b50-6b6c9fa4a773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:03:37.043767 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:37.043738    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-14af5c/simulator-w6-14af5c-kserve-85b8bd96d7-wlcb9" podUID="39766c7d-5f1d-42db-8920-c0321cd12c99" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Oct 30 22:03:37.050547 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:37.046965    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-14af5c/distinct-w6-14af5c-kserve-7c7cb9d69c-hljl9"
Oct 30 22:03:37.123522 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:37.122966    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-blrrb"
Oct 30 22:03:37.159760 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:37.158318    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-5be0d1/premium-w2-5be0d1-kserve-759b6fb995-c6gp4"
Oct 30 22:03:37.165759 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:37.165383    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2d1f91/premium-w0-2d1f91-kserve-6965b558cf-59lgw" podUID="fb84233d-25e0-48bc-8899-c3a3cb3cfeb9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 30 22:03:37.165915 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:37.165884    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-5be0d1/distinct-w2-5be0d1-kserve-6b85698c8-sxn9k"
Oct 30 22:03:37.169556 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:37.169515    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-17d809/simulator-w4-17d809-kserve-78d67b6f4-7hsvb"
Oct 30 22:03:37.170553 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:37.170519    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a07620/simulator-w1-a07620-kserve-55b44674f4-4z9lk"
Oct 30 22:03:37.207161 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:37.189799    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-14af5c/premium-w6-14af5c-kserve-6549647f4-f4gwg" podUID="9b42ee01-c60e-462c-a0e0-b04a52014db6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:03:37.217167 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:37.217144    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2d1f91/simulator-w0-2d1f91-kserve-7699765c74-jwtkb"
Oct 30 22:03:37.219275 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:37.219238    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-14af5c/premium-w6-14af5c-kserve-6549647f4-f4gwg"
Oct 30 22:03:37.223009 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:37.222984    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2d1f91/premium-w0-2d1f91-kserve-6965b558cf-59lgw"
Oct 30 22:03:37.223795 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:37.223398    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-14af5c/distinct-2-w6-14af5c-kserve-58d554bdb6-r5r6j"
Oct 30 22:03:37.224155 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:37.224135    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-17d809/premium-w4-17d809-kserve-7c9bdbbcbc-5mtmm"
Oct 30 22:03:37.229709 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:37.229547    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-17d809/distinct-w4-17d809-kserve-6b8f5c48cd-lhrwp" podUID="ef5e8691-42e1-471b-ba7d-90881028eca7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded"
Oct 30 22:03:37.242005 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:37.239157    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-14af5c/simulator-w6-14af5c-kserve-85b8bd96d7-wlcb9"
Oct 30 22:03:37.267226 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:37.267190    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-17d809/distinct-w4-17d809-kserve-6b8f5c48cd-lhrwp"
Oct 30 22:03:40.290414 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:40.290376    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-m4q9b"]
Oct 30 22:03:40.295018 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:40.294997    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-m4q9b"
Oct 30 22:03:40.364016 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:40.363982    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd9cb1c8-e190-4652-b11d-81299bbd7bcb-istiod-ca-cert\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-m4q9b\" (UID: \"fd9cb1c8-e190-4652-b11d-81299bbd7bcb\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-m4q9b"
Oct 30 22:03:40.364160 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:40.364112    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd9cb1c8-e190-4652-b11d-81299bbd7bcb-istio-token\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-m4q9b\" (UID: \"fd9cb1c8-e190-4652-b11d-81299bbd7bcb\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-m4q9b"
Oct 30 22:03:40.364207 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:40.364193    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kt2pk\" (UniqueName: \"kubernetes.io/projected/fd9cb1c8-e190-4652-b11d-81299bbd7bcb-kube-api-access-kt2pk\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-m4q9b\" (UID: \"fd9cb1c8-e190-4652-b11d-81299bbd7bcb\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-m4q9b"
Oct 30 22:03:40.364329 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:40.364313    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd9cb1c8-e190-4652-b11d-81299bbd7bcb-workload-certs\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-m4q9b\" (UID: \"fd9cb1c8-e190-4652-b11d-81299bbd7bcb\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-m4q9b"
Oct 30 22:03:40.364373 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:40.364350    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd9cb1c8-e190-4652-b11d-81299bbd7bcb-istio-envoy\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-m4q9b\" (UID: \"fd9cb1c8-e190-4652-b11d-81299bbd7bcb\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-m4q9b"
Oct 30 22:03:40.364492 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:40.364471    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd9cb1c8-e190-4652-b11d-81299bbd7bcb-istio-podinfo\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-m4q9b\" (UID: \"fd9cb1c8-e190-4652-b11d-81299bbd7bcb\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-m4q9b"
Oct 30 22:03:40.364582 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:40.364565    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd9cb1c8-e190-4652-b11d-81299bbd7bcb-istio-data\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-m4q9b\" (UID: \"fd9cb1c8-e190-4652-b11d-81299bbd7bcb\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-m4q9b"
Oct 30 22:03:40.364695 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:40.364680    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd9cb1c8-e190-4652-b11d-81299bbd7bcb-workload-socket\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-m4q9b\" (UID: \"fd9cb1c8-e190-4652-b11d-81299bbd7bcb\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-m4q9b"
Oct 30 22:03:40.364756 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:40.364724    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd9cb1c8-e190-4652-b11d-81299bbd7bcb-credential-socket\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-m4q9b\" (UID: \"fd9cb1c8-e190-4652-b11d-81299bbd7bcb\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-m4q9b"
Oct 30 22:03:40.383522 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:40.383494    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-m4q9b"]
Oct 30 22:03:40.466412 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:40.466380    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd9cb1c8-e190-4652-b11d-81299bbd7bcb-istio-podinfo\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-m4q9b\" (UID: \"fd9cb1c8-e190-4652-b11d-81299bbd7bcb\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-m4q9b"
Oct 30 22:03:40.467758 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:40.466435    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd9cb1c8-e190-4652-b11d-81299bbd7bcb-istio-data\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-m4q9b\" (UID: \"fd9cb1c8-e190-4652-b11d-81299bbd7bcb\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-m4q9b"
Oct 30 22:03:40.467758 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:40.466481    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd9cb1c8-e190-4652-b11d-81299bbd7bcb-workload-socket\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-m4q9b\" (UID: \"fd9cb1c8-e190-4652-b11d-81299bbd7bcb\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-m4q9b"
Oct 30 22:03:40.467758 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:40.466506    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd9cb1c8-e190-4652-b11d-81299bbd7bcb-credential-socket\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-m4q9b\" (UID: \"fd9cb1c8-e190-4652-b11d-81299bbd7bcb\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-m4q9b"
Oct 30 22:03:40.467758 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:40.466720    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd9cb1c8-e190-4652-b11d-81299bbd7bcb-istiod-ca-cert\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-m4q9b\" (UID: \"fd9cb1c8-e190-4652-b11d-81299bbd7bcb\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-m4q9b"
Oct 30 22:03:40.467758 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:40.466815    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd9cb1c8-e190-4652-b11d-81299bbd7bcb-istio-token\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-m4q9b\" (UID: \"fd9cb1c8-e190-4652-b11d-81299bbd7bcb\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-m4q9b"
Oct 30 22:03:40.467758 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:40.466871    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kt2pk\" (UniqueName: \"kubernetes.io/projected/fd9cb1c8-e190-4652-b11d-81299bbd7bcb-kube-api-access-kt2pk\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-m4q9b\" (UID: \"fd9cb1c8-e190-4652-b11d-81299bbd7bcb\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-m4q9b"
Oct 30 22:03:40.467758 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:40.466939    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd9cb1c8-e190-4652-b11d-81299bbd7bcb-workload-certs\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-m4q9b\" (UID: \"fd9cb1c8-e190-4652-b11d-81299bbd7bcb\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-m4q9b"
Oct 30 22:03:40.467758 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:40.466971    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd9cb1c8-e190-4652-b11d-81299bbd7bcb-istio-envoy\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-m4q9b\" (UID: \"fd9cb1c8-e190-4652-b11d-81299bbd7bcb\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-m4q9b"
Oct 30 22:03:40.467758 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:40.467013    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd9cb1c8-e190-4652-b11d-81299bbd7bcb-istio-data\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-m4q9b\" (UID: \"fd9cb1c8-e190-4652-b11d-81299bbd7bcb\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-m4q9b"
Oct 30 22:03:40.467758 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:40.467417    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd9cb1c8-e190-4652-b11d-81299bbd7bcb-workload-certs\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-m4q9b\" (UID: \"fd9cb1c8-e190-4652-b11d-81299bbd7bcb\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-m4q9b"
Oct 30 22:03:40.467758 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:40.467580    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd9cb1c8-e190-4652-b11d-81299bbd7bcb-credential-socket\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-m4q9b\" (UID: \"fd9cb1c8-e190-4652-b11d-81299bbd7bcb\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-m4q9b"
Oct 30 22:03:40.468393 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:40.468116    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd9cb1c8-e190-4652-b11d-81299bbd7bcb-istiod-ca-cert\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-m4q9b\" (UID: \"fd9cb1c8-e190-4652-b11d-81299bbd7bcb\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-m4q9b"
Oct 30 22:03:40.468393 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:40.468179    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd9cb1c8-e190-4652-b11d-81299bbd7bcb-workload-socket\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-m4q9b\" (UID: \"fd9cb1c8-e190-4652-b11d-81299bbd7bcb\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-m4q9b"
Oct 30 22:03:40.469368 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:40.469346    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd9cb1c8-e190-4652-b11d-81299bbd7bcb-istio-envoy\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-m4q9b\" (UID: \"fd9cb1c8-e190-4652-b11d-81299bbd7bcb\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-m4q9b"
Oct 30 22:03:40.469746 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:40.469727    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd9cb1c8-e190-4652-b11d-81299bbd7bcb-istio-podinfo\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-m4q9b\" (UID: \"fd9cb1c8-e190-4652-b11d-81299bbd7bcb\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-m4q9b"
Oct 30 22:03:40.522320 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:40.522286    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd9cb1c8-e190-4652-b11d-81299bbd7bcb-istio-token\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-m4q9b\" (UID: \"fd9cb1c8-e190-4652-b11d-81299bbd7bcb\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-m4q9b"
Oct 30 22:03:40.522439 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:40.522418    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kt2pk\" (UniqueName: \"kubernetes.io/projected/fd9cb1c8-e190-4652-b11d-81299bbd7bcb-kube-api-access-kt2pk\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-m4q9b\" (UID: \"fd9cb1c8-e190-4652-b11d-81299bbd7bcb\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-m4q9b"
Oct 30 22:03:40.609889 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:40.609862    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-m4q9b"
Oct 30 22:03:41.057881 ip-10-0-130-60 kubenswrapper[5061]: W1030 22:03:41.057845    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd9cb1c8_e190_4652_b11d_81299bbd7bcb.slice/crio-244e4504cc3374316f081bcc38cba4b74a3760a556d2f4a4406cdc2e28d6d55f WatchSource:0}: Error finding container 244e4504cc3374316f081bcc38cba4b74a3760a556d2f4a4406cdc2e28d6d55f: Status 404 returned error can't find the container with id 244e4504cc3374316f081bcc38cba4b74a3760a556d2f4a4406cdc2e28d6d55f
Oct 30 22:03:41.058109 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:41.058081    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-m4q9b"]
Oct 30 22:03:41.060510 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:41.060482    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:03:41.060607 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:41.060560    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:03:41.060607 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:41.060591    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:03:41.543086 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:41.543007    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-m4q9b" event={"ID":"fd9cb1c8-e190-4652-b11d-81299bbd7bcb","Type":"ContainerStarted","Data":"9ac966264097d395670ddfdde2337c93ce57769db01bed0c7e5ff4ae9f502d07"}
Oct 30 22:03:41.543086 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:41.543053    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-m4q9b" event={"ID":"fd9cb1c8-e190-4652-b11d-81299bbd7bcb","Type":"ContainerStarted","Data":"244e4504cc3374316f081bcc38cba4b74a3760a556d2f4a4406cdc2e28d6d55f"}
Oct 30 22:03:41.610728 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:41.610695    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-m4q9b"
Oct 30 22:03:41.658934 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:41.658888    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-m4q9b" podStartSLOduration=1.6588724620000002 podStartE2EDuration="1.658872462s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:03:41.657474611 +0000 UTC m=+1275.270244380" watchObservedRunningTime="2026-10-30 22:03:41.658872462 +0000 UTC m=+1275.271642219"
Oct 30 22:03:42.634376 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:42.634342    5061 patch_prober.go:29] interesting pod/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-m4q9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Oct 30 22:03:42.634843 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:42.634397    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-m4q9b" podUID="fd9cb1c8-e190-4652-b11d-81299bbd7bcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Oct 30 22:03:43.629553 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:43.618846    5061 patch_prober.go:29] interesting pod/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-m4q9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Oct 30 22:03:43.629553 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:43.618905    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-m4q9b" podUID="fd9cb1c8-e190-4652-b11d-81299bbd7bcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Oct 30 22:03:44.632967 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:44.632922    5061 patch_prober.go:29] interesting pod/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-m4q9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Oct 30 22:03:44.633357 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:44.632988    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-m4q9b" podUID="fd9cb1c8-e190-4652-b11d-81299bbd7bcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Oct 30 22:03:45.628549 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:45.627859    5061 patch_prober.go:29] interesting pod/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-m4q9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Oct 30 22:03:45.628549 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:45.627915    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-m4q9b" podUID="fd9cb1c8-e190-4652-b11d-81299bbd7bcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Oct 30 22:03:46.632844 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:46.628870    5061 patch_prober.go:29] interesting pod/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-m4q9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Oct 30 22:03:46.632844 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:46.628959    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-m4q9b" podUID="fd9cb1c8-e190-4652-b11d-81299bbd7bcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Oct 30 22:03:47.610710 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:47.610677    5061 patch_prober.go:29] interesting pod/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-m4q9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Oct 30 22:03:47.610911 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:47.610725    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-m4q9b" podUID="fd9cb1c8-e190-4652-b11d-81299bbd7bcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Oct 30 22:03:48.639468 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:48.639204    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-m4q9b"
Oct 30 22:03:48.640015 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:48.639724    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-m4q9b"
Oct 30 22:03:48.641193 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:48.641165    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-m4q9b"
Oct 30 22:03:49.283017 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:49.282978    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8123afac-hn7mk"]
Oct 30 22:03:49.287894 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:49.287867    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8123afac-hn7mk"
Oct 30 22:03:49.337682 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:49.337650    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:03:49.337822 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:49.337788    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gdmfg\""
Oct 30 22:03:49.344671 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:49.344644    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8123afac-hn7mk"]
Oct 30 22:03:49.360675 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:49.360647    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8gk7\" (UniqueName: \"kubernetes.io/projected/79096c68-4183-43fc-8262-2668e2f26057-kube-api-access-l8gk7\") pod \"maas-api-revoke-sub-8123afac-hn7mk\" (UID: \"79096c68-4183-43fc-8262-2668e2f26057\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8123afac-hn7mk"
Oct 30 22:03:49.360888 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:49.360861    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79096c68-4183-43fc-8262-2668e2f26057-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8123afac-hn7mk\" (UID: \"79096c68-4183-43fc-8262-2668e2f26057\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8123afac-hn7mk"
Oct 30 22:03:49.361001 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:49.360980    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79096c68-4183-43fc-8262-2668e2f26057-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8123afac-hn7mk\" (UID: \"79096c68-4183-43fc-8262-2668e2f26057\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8123afac-hn7mk"
Oct 30 22:03:49.462072 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:49.462033    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79096c68-4183-43fc-8262-2668e2f26057-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8123afac-hn7mk\" (UID: \"79096c68-4183-43fc-8262-2668e2f26057\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8123afac-hn7mk"
Oct 30 22:03:49.462285 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:49.462263    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8gk7\" (UniqueName: \"kubernetes.io/projected/79096c68-4183-43fc-8262-2668e2f26057-kube-api-access-l8gk7\") pod \"maas-api-revoke-sub-8123afac-hn7mk\" (UID: \"79096c68-4183-43fc-8262-2668e2f26057\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8123afac-hn7mk"
Oct 30 22:03:49.462511 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:49.462480    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79096c68-4183-43fc-8262-2668e2f26057-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8123afac-hn7mk\" (UID: \"79096c68-4183-43fc-8262-2668e2f26057\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8123afac-hn7mk"
Oct 30 22:03:49.462960 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:49.462934    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79096c68-4183-43fc-8262-2668e2f26057-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8123afac-hn7mk\" (UID: \"79096c68-4183-43fc-8262-2668e2f26057\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8123afac-hn7mk"
Oct 30 22:03:49.465062 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:49.465037    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79096c68-4183-43fc-8262-2668e2f26057-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8123afac-hn7mk\" (UID: \"79096c68-4183-43fc-8262-2668e2f26057\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8123afac-hn7mk"
Oct 30 22:03:49.520290 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:49.520257    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8gk7\" (UniqueName: \"kubernetes.io/projected/79096c68-4183-43fc-8262-2668e2f26057-kube-api-access-l8gk7\") pod \"maas-api-revoke-sub-8123afac-hn7mk\" (UID: \"79096c68-4183-43fc-8262-2668e2f26057\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8123afac-hn7mk"
Oct 30 22:03:49.603149 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:49.603120    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8123afac-hn7mk"
Oct 30 22:03:49.992014 ip-10-0-130-60 kubenswrapper[5061]: W1030 22:03:49.991975    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79096c68_4183_43fc_8262_2668e2f26057.slice/crio-66c339d885ae7bac3b429fe71ddc020bee40b4bf6b9ae135e3a5ec838c701aca WatchSource:0}: Error finding container 66c339d885ae7bac3b429fe71ddc020bee40b4bf6b9ae135e3a5ec838c701aca: Status 404 returned error can't find the container with id 66c339d885ae7bac3b429fe71ddc020bee40b4bf6b9ae135e3a5ec838c701aca
Oct 30 22:03:49.993281 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:49.993254    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8123afac-hn7mk"]
Oct 30 22:03:50.609824 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:50.609784    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8123afac-hn7mk" event={"ID":"79096c68-4183-43fc-8262-2668e2f26057","Type":"ContainerStarted","Data":"1ecd168e1a486e07197222219256636047ce3ed4ee26a9edfa7b8b602101a9f5"}
Oct 30 22:03:50.609989 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:50.609850    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8123afac-hn7mk" event={"ID":"79096c68-4183-43fc-8262-2668e2f26057","Type":"ContainerStarted","Data":"66c339d885ae7bac3b429fe71ddc020bee40b4bf6b9ae135e3a5ec838c701aca"}
Oct 30 22:03:50.719646 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:50.719549    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8123afac-hn7mk" podStartSLOduration=1.719515592 podStartE2EDuration="1.719515592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:03:50.718497479 +0000 UTC m=+1284.331267271" watchObservedRunningTime="2026-10-30 22:03:50.719515592 +0000 UTC m=+1284.332285349"
Oct 30 22:03:51.618189 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:51.618155    5061 generic.go:356] "Generic (PLEG): container finished" podID="79096c68-4183-43fc-8262-2668e2f26057" containerID="1ecd168e1a486e07197222219256636047ce3ed4ee26a9edfa7b8b602101a9f5" exitCode=0
Oct 30 22:03:51.618678 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:51.618237    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8123afac-hn7mk" event={"ID":"79096c68-4183-43fc-8262-2668e2f26057","Type":"ContainerDied","Data":"1ecd168e1a486e07197222219256636047ce3ed4ee26a9edfa7b8b602101a9f5"}
Oct 30 22:03:52.792452 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:52.792425    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8123afac-hn7mk"
Oct 30 22:03:52.912251 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:52.912181    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79096c68-4183-43fc-8262-2668e2f26057-kube-api-access-l8gk7\" (UniqueName: \"kubernetes.io/projected/79096c68-4183-43fc-8262-2668e2f26057-kube-api-access-l8gk7\") pod \"79096c68-4183-43fc-8262-2668e2f26057\" (UID: \"79096c68-4183-43fc-8262-2668e2f26057\") "
Oct 30 22:03:52.912385 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:52.912251    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79096c68-4183-43fc-8262-2668e2f26057-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79096c68-4183-43fc-8262-2668e2f26057-maas-api-openshift-ca\") pod \"79096c68-4183-43fc-8262-2668e2f26057\" (UID: \"79096c68-4183-43fc-8262-2668e2f26057\") "
Oct 30 22:03:52.912449 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:52.912432    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79096c68-4183-43fc-8262-2668e2f26057-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79096c68-4183-43fc-8262-2668e2f26057-maas-api-xks-ca\") pod \"79096c68-4183-43fc-8262-2668e2f26057\" (UID: \"79096c68-4183-43fc-8262-2668e2f26057\") "
Oct 30 22:03:52.912724 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:52.912699    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79096c68-4183-43fc-8262-2668e2f26057-maas-api-openshift-ca" pod "79096c68-4183-43fc-8262-2668e2f26057" (UID: "79096c68-4183-43fc-8262-2668e2f26057"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:03:52.913105 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:52.913082    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79096c68-4183-43fc-8262-2668e2f26057-maas-api-openshift-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:52.915585 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:52.915560    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79096c68-4183-43fc-8262-2668e2f26057-maas-api-xks-ca" pod "79096c68-4183-43fc-8262-2668e2f26057" (UID: "79096c68-4183-43fc-8262-2668e2f26057"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:03:52.915672 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:52.915630    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79096c68-4183-43fc-8262-2668e2f26057-kube-api-access-l8gk7" pod "79096c68-4183-43fc-8262-2668e2f26057" (UID: "79096c68-4183-43fc-8262-2668e2f26057"). InnerVolumeSpecName "kube-api-access-l8gk7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:03:53.014085 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:53.014054    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79096c68-4183-43fc-8262-2668e2f26057-maas-api-xks-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:53.014085 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:53.014084    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8gk7\" (UniqueName: \"kubernetes.io/projected/79096c68-4183-43fc-8262-2668e2f26057-kube-api-access-l8gk7\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:53.364592 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:53.364561    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gg4hd"]
Oct 30 22:03:53.364791 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:53.364759    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gg4hd" podUID="fc913c19-cac2-4d4e-93fb-06fddd58ed78" containerName="istio-proxy" containerID="cri-o://866afe08bd97e1bcfcdf42a15293ecc14d98de6c1951e2a2a30d7fe24a97bccb" gracePeriod=30
Oct 30 22:03:53.411702 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:53.411667    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-znswb"]
Oct 30 22:03:53.411949 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:53.411922    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-znswb" podUID="e8cf2cfc-250e-47a7-a8ef-25f6e343afd8" containerName="istio-proxy" containerID="cri-o://409f68272498f576ec2ea40a91a3f64d337a7cee24bcd68b568911882de9d529" gracePeriod=30
Oct 30 22:03:53.416257 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:53.416228    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4fx5l"]
Oct 30 22:03:53.416421 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:53.416401    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4fx5l" podUID="c55fb717-8e3a-4c50-b0e7-249115db4dda" containerName="istio-proxy" containerID="cri-o://ff20afd34720205e27cecdf86435ddb1ba57242f30d0310531184f407f2a3fab" gracePeriod=30
Oct 30 22:03:53.633766 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:53.633689    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8123afac-hn7mk"
Oct 30 22:03:53.633766 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:53.633722    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8123afac-hn7mk" event={"ID":"79096c68-4183-43fc-8262-2668e2f26057","Type":"ContainerDied","Data":"66c339d885ae7bac3b429fe71ddc020bee40b4bf6b9ae135e3a5ec838c701aca"}
Oct 30 22:03:53.633981 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:53.633774    5061 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="66c339d885ae7bac3b429fe71ddc020bee40b4bf6b9ae135e3a5ec838c701aca"
Oct 30 22:03:54.668636 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:03:54.668601    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 22:03:54.716116 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:54.716085    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-5z645"]
Oct 30 22:03:54.716822 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:54.716803    5061 cpu_manager.go:414] "containerMap: removing container" podUID="79096c68-4183-43fc-8262-2668e2f26057" containerName="revoke-keys"
Oct 30 22:03:54.716822 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:54.716821    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79096c68-4183-43fc-8262-2668e2f26057" containerName="revoke-keys"
Oct 30 22:03:54.716956 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:54.716937    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="79096c68-4183-43fc-8262-2668e2f26057" containerName="revoke-keys"
Oct 30 22:03:54.720958 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:54.720940    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-5z645"
Oct 30 22:03:54.829015 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:54.828983    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-5z645"]
Oct 30 22:03:54.840494 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:54.840462    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a84e4890-ba61-49a5-b9b6-be9cf741d2d8-istio-data\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-5z645\" (UID: \"a84e4890-ba61-49a5-b9b6-be9cf741d2d8\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-5z645"
Oct 30 22:03:54.840674 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:54.840519    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a84e4890-ba61-49a5-b9b6-be9cf741d2d8-credential-socket\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-5z645\" (UID: \"a84e4890-ba61-49a5-b9b6-be9cf741d2d8\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-5z645"
Oct 30 22:03:54.840756 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:54.840672    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a84e4890-ba61-49a5-b9b6-be9cf741d2d8-workload-certs\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-5z645\" (UID: \"a84e4890-ba61-49a5-b9b6-be9cf741d2d8\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-5z645"
Oct 30 22:03:54.840756 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:54.840716    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a84e4890-ba61-49a5-b9b6-be9cf741d2d8-istio-podinfo\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-5z645\" (UID: \"a84e4890-ba61-49a5-b9b6-be9cf741d2d8\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-5z645"
Oct 30 22:03:54.840866 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:54.840850    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a84e4890-ba61-49a5-b9b6-be9cf741d2d8-workload-socket\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-5z645\" (UID: \"a84e4890-ba61-49a5-b9b6-be9cf741d2d8\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-5z645"
Oct 30 22:03:54.840921 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:54.840886    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a84e4890-ba61-49a5-b9b6-be9cf741d2d8-istio-envoy\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-5z645\" (UID: \"a84e4890-ba61-49a5-b9b6-be9cf741d2d8\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-5z645"
Oct 30 22:03:54.841038 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:54.841022    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfm9k\" (UniqueName: \"kubernetes.io/projected/a84e4890-ba61-49a5-b9b6-be9cf741d2d8-kube-api-access-mfm9k\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-5z645\" (UID: \"a84e4890-ba61-49a5-b9b6-be9cf741d2d8\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-5z645"
Oct 30 22:03:54.841097 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:54.841053    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a84e4890-ba61-49a5-b9b6-be9cf741d2d8-istiod-ca-cert\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-5z645\" (UID: \"a84e4890-ba61-49a5-b9b6-be9cf741d2d8\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-5z645"
Oct 30 22:03:54.841218 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:54.841202    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a84e4890-ba61-49a5-b9b6-be9cf741d2d8-istio-token\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-5z645\" (UID: \"a84e4890-ba61-49a5-b9b6-be9cf741d2d8\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-5z645"
Oct 30 22:03:54.942469 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:54.942375    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfm9k\" (UniqueName: \"kubernetes.io/projected/a84e4890-ba61-49a5-b9b6-be9cf741d2d8-kube-api-access-mfm9k\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-5z645\" (UID: \"a84e4890-ba61-49a5-b9b6-be9cf741d2d8\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-5z645"
Oct 30 22:03:54.942469 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:54.942423    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a84e4890-ba61-49a5-b9b6-be9cf741d2d8-istiod-ca-cert\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-5z645\" (UID: \"a84e4890-ba61-49a5-b9b6-be9cf741d2d8\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-5z645"
Oct 30 22:03:54.942747 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:54.942516    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a84e4890-ba61-49a5-b9b6-be9cf741d2d8-istio-token\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-5z645\" (UID: \"a84e4890-ba61-49a5-b9b6-be9cf741d2d8\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-5z645"
Oct 30 22:03:54.942747 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:54.942625    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a84e4890-ba61-49a5-b9b6-be9cf741d2d8-istio-data\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-5z645\" (UID: \"a84e4890-ba61-49a5-b9b6-be9cf741d2d8\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-5z645"
Oct 30 22:03:54.942747 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:54.942665    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a84e4890-ba61-49a5-b9b6-be9cf741d2d8-credential-socket\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-5z645\" (UID: \"a84e4890-ba61-49a5-b9b6-be9cf741d2d8\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-5z645"
Oct 30 22:03:54.942906 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:54.942753    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a84e4890-ba61-49a5-b9b6-be9cf741d2d8-workload-certs\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-5z645\" (UID: \"a84e4890-ba61-49a5-b9b6-be9cf741d2d8\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-5z645"
Oct 30 22:03:54.942906 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:54.942785    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a84e4890-ba61-49a5-b9b6-be9cf741d2d8-istio-podinfo\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-5z645\" (UID: \"a84e4890-ba61-49a5-b9b6-be9cf741d2d8\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-5z645"
Oct 30 22:03:54.942906 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:54.942876    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a84e4890-ba61-49a5-b9b6-be9cf741d2d8-workload-socket\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-5z645\" (UID: \"a84e4890-ba61-49a5-b9b6-be9cf741d2d8\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-5z645"
Oct 30 22:03:54.943050 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:54.942909    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a84e4890-ba61-49a5-b9b6-be9cf741d2d8-istio-envoy\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-5z645\" (UID: \"a84e4890-ba61-49a5-b9b6-be9cf741d2d8\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-5z645"
Oct 30 22:03:54.943107 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:54.943062    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a84e4890-ba61-49a5-b9b6-be9cf741d2d8-istio-data\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-5z645\" (UID: \"a84e4890-ba61-49a5-b9b6-be9cf741d2d8\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-5z645"
Oct 30 22:03:54.943159 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:54.943113    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a84e4890-ba61-49a5-b9b6-be9cf741d2d8-credential-socket\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-5z645\" (UID: \"a84e4890-ba61-49a5-b9b6-be9cf741d2d8\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-5z645"
Oct 30 22:03:54.943274 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:54.943255    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a84e4890-ba61-49a5-b9b6-be9cf741d2d8-workload-certs\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-5z645\" (UID: \"a84e4890-ba61-49a5-b9b6-be9cf741d2d8\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-5z645"
Oct 30 22:03:54.943342 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:54.943291    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a84e4890-ba61-49a5-b9b6-be9cf741d2d8-istiod-ca-cert\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-5z645\" (UID: \"a84e4890-ba61-49a5-b9b6-be9cf741d2d8\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-5z645"
Oct 30 22:03:54.943342 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:54.943321    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a84e4890-ba61-49a5-b9b6-be9cf741d2d8-workload-socket\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-5z645\" (UID: \"a84e4890-ba61-49a5-b9b6-be9cf741d2d8\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-5z645"
Oct 30 22:03:54.945357 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:54.945328    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a84e4890-ba61-49a5-b9b6-be9cf741d2d8-istio-envoy\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-5z645\" (UID: \"a84e4890-ba61-49a5-b9b6-be9cf741d2d8\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-5z645"
Oct 30 22:03:54.945484 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:54.945464    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a84e4890-ba61-49a5-b9b6-be9cf741d2d8-istio-podinfo\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-5z645\" (UID: \"a84e4890-ba61-49a5-b9b6-be9cf741d2d8\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-5z645"
Oct 30 22:03:54.999062 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:54.999031    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfm9k\" (UniqueName: \"kubernetes.io/projected/a84e4890-ba61-49a5-b9b6-be9cf741d2d8-kube-api-access-mfm9k\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-5z645\" (UID: \"a84e4890-ba61-49a5-b9b6-be9cf741d2d8\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-5z645"
Oct 30 22:03:54.999283 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:54.999265    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a84e4890-ba61-49a5-b9b6-be9cf741d2d8-istio-token\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-5z645\" (UID: \"a84e4890-ba61-49a5-b9b6-be9cf741d2d8\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-5z645"
Oct 30 22:03:55.035253 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:55.035228    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-5z645"
Oct 30 22:03:55.461026 ip-10-0-130-60 kubenswrapper[5061]: W1030 22:03:55.460992    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda84e4890_ba61_49a5_b9b6_be9cf741d2d8.slice/crio-834b6990ef43d6b1085f57128fcb5f4446c48d344d632e8bea6abd57f92d24b7 WatchSource:0}: Error finding container 834b6990ef43d6b1085f57128fcb5f4446c48d344d632e8bea6abd57f92d24b7: Status 404 returned error can't find the container with id 834b6990ef43d6b1085f57128fcb5f4446c48d344d632e8bea6abd57f92d24b7
Oct 30 22:03:55.462084 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:55.462054    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-5z645"]
Oct 30 22:03:55.463884 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:55.463843    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:03:55.463990 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:55.463921    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:03:55.463990 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:55.463970    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:03:55.650542 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:55.650501    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-5z645" event={"ID":"a84e4890-ba61-49a5-b9b6-be9cf741d2d8","Type":"ContainerStarted","Data":"b4886f646e7e94afb16aff7aa126959dd64ed13c753ca92a3013d38214007232"}
Oct 30 22:03:55.650694 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:55.650578    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-5z645" event={"ID":"a84e4890-ba61-49a5-b9b6-be9cf741d2d8","Type":"ContainerStarted","Data":"834b6990ef43d6b1085f57128fcb5f4446c48d344d632e8bea6abd57f92d24b7"}
Oct 30 22:03:55.778058 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:55.777966    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-5z645" podStartSLOduration=1.777952257 podStartE2EDuration="1.777952257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:03:55.777001898 +0000 UTC m=+1289.389771656" watchObservedRunningTime="2026-10-30 22:03:55.777952257 +0000 UTC m=+1289.390722015"
Oct 30 22:03:56.036420 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:56.036243    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-5z645"
Oct 30 22:03:56.038037 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:56.037998    5061 patch_prober.go:29] interesting pod/e2e-worker-w3-f776cb-openshift-default-74d577586f-5z645 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Oct 30 22:03:56.038217 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:56.038067    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-5z645" podUID="a84e4890-ba61-49a5-b9b6-be9cf741d2d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Oct 30 22:03:56.137553 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:56.136879    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-kmtkh"]
Oct 30 22:03:56.141742 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:56.141719    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-kmtkh"
Oct 30 22:03:56.196770 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:56.196742    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-kmtkh"]
Oct 30 22:03:56.266075 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:56.266043    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03533dfe-8974-487d-9af6-4adda6fc8d8b-istio-envoy\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-kmtkh\" (UID: \"03533dfe-8974-487d-9af6-4adda6fc8d8b\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-kmtkh"
Oct 30 22:03:56.266299 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:56.266182    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vw74n\" (UniqueName: \"kubernetes.io/projected/03533dfe-8974-487d-9af6-4adda6fc8d8b-kube-api-access-vw74n\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-kmtkh\" (UID: \"03533dfe-8974-487d-9af6-4adda6fc8d8b\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-kmtkh"
Oct 30 22:03:56.266299 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:56.266229    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03533dfe-8974-487d-9af6-4adda6fc8d8b-istiod-ca-cert\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-kmtkh\" (UID: \"03533dfe-8974-487d-9af6-4adda6fc8d8b\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-kmtkh"
Oct 30 22:03:56.266299 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:56.266258    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03533dfe-8974-487d-9af6-4adda6fc8d8b-workload-certs\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-kmtkh\" (UID: \"03533dfe-8974-487d-9af6-4adda6fc8d8b\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-kmtkh"
Oct 30 22:03:56.266405 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:56.266304    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03533dfe-8974-487d-9af6-4adda6fc8d8b-istio-data\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-kmtkh\" (UID: \"03533dfe-8974-487d-9af6-4adda6fc8d8b\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-kmtkh"
Oct 30 22:03:56.266443 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:56.266399    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03533dfe-8974-487d-9af6-4adda6fc8d8b-istio-podinfo\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-kmtkh\" (UID: \"03533dfe-8974-487d-9af6-4adda6fc8d8b\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-kmtkh"
Oct 30 22:03:56.266477 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:56.266440    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03533dfe-8974-487d-9af6-4adda6fc8d8b-credential-socket\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-kmtkh\" (UID: \"03533dfe-8974-487d-9af6-4adda6fc8d8b\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-kmtkh"
Oct 30 22:03:56.266507 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:56.266490    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03533dfe-8974-487d-9af6-4adda6fc8d8b-workload-socket\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-kmtkh\" (UID: \"03533dfe-8974-487d-9af6-4adda6fc8d8b\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-kmtkh"
Oct 30 22:03:56.266573 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:56.266554    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03533dfe-8974-487d-9af6-4adda6fc8d8b-istio-token\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-kmtkh\" (UID: \"03533dfe-8974-487d-9af6-4adda6fc8d8b\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-kmtkh"
Oct 30 22:03:56.368626 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:56.368590    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03533dfe-8974-487d-9af6-4adda6fc8d8b-istio-envoy\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-kmtkh\" (UID: \"03533dfe-8974-487d-9af6-4adda6fc8d8b\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-kmtkh"
Oct 30 22:03:56.368817 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:56.368684    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vw74n\" (UniqueName: \"kubernetes.io/projected/03533dfe-8974-487d-9af6-4adda6fc8d8b-kube-api-access-vw74n\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-kmtkh\" (UID: \"03533dfe-8974-487d-9af6-4adda6fc8d8b\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-kmtkh"
Oct 30 22:03:56.368817 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:56.368704    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03533dfe-8974-487d-9af6-4adda6fc8d8b-istiod-ca-cert\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-kmtkh\" (UID: \"03533dfe-8974-487d-9af6-4adda6fc8d8b\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-kmtkh"
Oct 30 22:03:56.368817 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:56.368731    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03533dfe-8974-487d-9af6-4adda6fc8d8b-workload-certs\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-kmtkh\" (UID: \"03533dfe-8974-487d-9af6-4adda6fc8d8b\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-kmtkh"
Oct 30 22:03:56.368817 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:56.368758    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03533dfe-8974-487d-9af6-4adda6fc8d8b-istio-data\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-kmtkh\" (UID: \"03533dfe-8974-487d-9af6-4adda6fc8d8b\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-kmtkh"
Oct 30 22:03:56.369042 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:56.368889    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03533dfe-8974-487d-9af6-4adda6fc8d8b-istio-podinfo\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-kmtkh\" (UID: \"03533dfe-8974-487d-9af6-4adda6fc8d8b\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-kmtkh"
Oct 30 22:03:56.369042 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:56.368962    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03533dfe-8974-487d-9af6-4adda6fc8d8b-credential-socket\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-kmtkh\" (UID: \"03533dfe-8974-487d-9af6-4adda6fc8d8b\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-kmtkh"
Oct 30 22:03:56.369042 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:56.369016    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03533dfe-8974-487d-9af6-4adda6fc8d8b-workload-socket\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-kmtkh\" (UID: \"03533dfe-8974-487d-9af6-4adda6fc8d8b\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-kmtkh"
Oct 30 22:03:56.369190 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:56.369069    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03533dfe-8974-487d-9af6-4adda6fc8d8b-istio-token\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-kmtkh\" (UID: \"03533dfe-8974-487d-9af6-4adda6fc8d8b\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-kmtkh"
Oct 30 22:03:56.369373 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:56.369350    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03533dfe-8974-487d-9af6-4adda6fc8d8b-istio-data\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-kmtkh\" (UID: \"03533dfe-8974-487d-9af6-4adda6fc8d8b\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-kmtkh"
Oct 30 22:03:56.369476 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:56.369415    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03533dfe-8974-487d-9af6-4adda6fc8d8b-workload-certs\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-kmtkh\" (UID: \"03533dfe-8974-487d-9af6-4adda6fc8d8b\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-kmtkh"
Oct 30 22:03:56.369476 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:56.369432    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03533dfe-8974-487d-9af6-4adda6fc8d8b-credential-socket\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-kmtkh\" (UID: \"03533dfe-8974-487d-9af6-4adda6fc8d8b\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-kmtkh"
Oct 30 22:03:56.369627 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:56.369515    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03533dfe-8974-487d-9af6-4adda6fc8d8b-istiod-ca-cert\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-kmtkh\" (UID: \"03533dfe-8974-487d-9af6-4adda6fc8d8b\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-kmtkh"
Oct 30 22:03:56.369627 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:56.369575    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03533dfe-8974-487d-9af6-4adda6fc8d8b-workload-socket\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-kmtkh\" (UID: \"03533dfe-8974-487d-9af6-4adda6fc8d8b\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-kmtkh"
Oct 30 22:03:56.371071 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:56.371052    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03533dfe-8974-487d-9af6-4adda6fc8d8b-istio-envoy\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-kmtkh\" (UID: \"03533dfe-8974-487d-9af6-4adda6fc8d8b\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-kmtkh"
Oct 30 22:03:56.371521 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:56.371503    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03533dfe-8974-487d-9af6-4adda6fc8d8b-istio-podinfo\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-kmtkh\" (UID: \"03533dfe-8974-487d-9af6-4adda6fc8d8b\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-kmtkh"
Oct 30 22:03:56.434809 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:56.434772    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03533dfe-8974-487d-9af6-4adda6fc8d8b-istio-token\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-kmtkh\" (UID: \"03533dfe-8974-487d-9af6-4adda6fc8d8b\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-kmtkh"
Oct 30 22:03:56.435154 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:56.435133    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vw74n\" (UniqueName: \"kubernetes.io/projected/03533dfe-8974-487d-9af6-4adda6fc8d8b-kube-api-access-vw74n\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-kmtkh\" (UID: \"03533dfe-8974-487d-9af6-4adda6fc8d8b\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-kmtkh"
Oct 30 22:03:56.458004 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:56.457977    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-kmtkh"
Oct 30 22:03:56.941879 ip-10-0-130-60 kubenswrapper[5061]: W1030 22:03:56.941834    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03533dfe_8974_487d_9af6_4adda6fc8d8b.slice/crio-5294fed5816bfc7761f8b107540477c1ee2bdae97896cac1f8d9ca308a2e9a1e WatchSource:0}: Error finding container 5294fed5816bfc7761f8b107540477c1ee2bdae97896cac1f8d9ca308a2e9a1e: Status 404 returned error can't find the container with id 5294fed5816bfc7761f8b107540477c1ee2bdae97896cac1f8d9ca308a2e9a1e
Oct 30 22:03:56.943548 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:56.943505    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-kmtkh"]
Oct 30 22:03:56.954318 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:56.950873    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:03:56.954318 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:56.951019    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:03:56.954318 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:56.951097    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:03:57.037418 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:57.037365    5061 patch_prober.go:29] interesting pod/e2e-worker-w3-f776cb-openshift-default-74d577586f-5z645 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Oct 30 22:03:57.037615 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:57.037443    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-5z645" podUID="a84e4890-ba61-49a5-b9b6-be9cf741d2d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Oct 30 22:03:57.674173 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:57.674126    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-kmtkh" event={"ID":"03533dfe-8974-487d-9af6-4adda6fc8d8b","Type":"ContainerStarted","Data":"49633a346981edd93f36051e1be89b70e4aed785827492ede902087003443483"}
Oct 30 22:03:57.674361 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:57.674267    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-kmtkh" event={"ID":"03533dfe-8974-487d-9af6-4adda6fc8d8b","Type":"ContainerStarted","Data":"5294fed5816bfc7761f8b107540477c1ee2bdae97896cac1f8d9ca308a2e9a1e"}
Oct 30 22:03:57.855610 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:57.847922    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-kmtkh" podStartSLOduration=1.847904481 podStartE2EDuration="1.847904481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:03:57.84290637 +0000 UTC m=+1291.455676142" watchObservedRunningTime="2026-10-30 22:03:57.847904481 +0000 UTC m=+1291.460674239"
Oct 30 22:03:58.047493 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:58.047270    5061 patch_prober.go:29] interesting pod/e2e-worker-w3-f776cb-openshift-default-74d577586f-5z645 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Oct 30 22:03:58.047493 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:58.047328    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-5z645" podUID="a84e4890-ba61-49a5-b9b6-be9cf741d2d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Oct 30 22:03:58.469898 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:58.460155    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-kmtkh"
Oct 30 22:03:58.469898 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:58.469676    5061 patch_prober.go:29] interesting pod/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-kmtkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Oct 30 22:03:58.469898 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:58.469768    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-kmtkh" podUID="03533dfe-8974-487d-9af6-4adda6fc8d8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Oct 30 22:03:58.732569 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:58.732171    5061 generic.go:356] "Generic (PLEG): container finished" podID="e8cf2cfc-250e-47a7-a8ef-25f6e343afd8" containerID="409f68272498f576ec2ea40a91a3f64d337a7cee24bcd68b568911882de9d529" exitCode=0
Oct 30 22:03:58.732569 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:58.732342    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-znswb" event={"ID":"e8cf2cfc-250e-47a7-a8ef-25f6e343afd8","Type":"ContainerDied","Data":"409f68272498f576ec2ea40a91a3f64d337a7cee24bcd68b568911882de9d529"}
Oct 30 22:03:58.798569 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:58.797452    5061 generic.go:356] "Generic (PLEG): container finished" podID="c55fb717-8e3a-4c50-b0e7-249115db4dda" containerID="ff20afd34720205e27cecdf86435ddb1ba57242f30d0310531184f407f2a3fab" exitCode=0
Oct 30 22:03:58.798569 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:58.797693    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4fx5l" event={"ID":"c55fb717-8e3a-4c50-b0e7-249115db4dda","Type":"ContainerDied","Data":"ff20afd34720205e27cecdf86435ddb1ba57242f30d0310531184f407f2a3fab"}
Oct 30 22:03:58.804171 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:58.802717    5061 generic.go:356] "Generic (PLEG): container finished" podID="fc913c19-cac2-4d4e-93fb-06fddd58ed78" containerID="866afe08bd97e1bcfcdf42a15293ecc14d98de6c1951e2a2a30d7fe24a97bccb" exitCode=0
Oct 30 22:03:58.804171 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:58.803189    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gg4hd" event={"ID":"fc913c19-cac2-4d4e-93fb-06fddd58ed78","Type":"ContainerDied","Data":"866afe08bd97e1bcfcdf42a15293ecc14d98de6c1951e2a2a30d7fe24a97bccb"}
Oct 30 22:03:58.979134 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:58.979047    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gg4hd"
Oct 30 22:03:59.067552 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:59.060038    5061 patch_prober.go:29] interesting pod/e2e-worker-w3-f776cb-openshift-default-74d577586f-5z645 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Oct 30 22:03:59.067552 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:59.060099    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-5z645" podUID="a84e4890-ba61-49a5-b9b6-be9cf741d2d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Oct 30 22:03:59.091722 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:59.078172    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc913c19-cac2-4d4e-93fb-06fddd58ed78-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc913c19-cac2-4d4e-93fb-06fddd58ed78-istio-envoy\") pod \"fc913c19-cac2-4d4e-93fb-06fddd58ed78\" (UID: \"fc913c19-cac2-4d4e-93fb-06fddd58ed78\") "
Oct 30 22:03:59.091722 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:59.078253    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc913c19-cac2-4d4e-93fb-06fddd58ed78-kube-api-access-gqdp8\" (UniqueName: \"kubernetes.io/projected/fc913c19-cac2-4d4e-93fb-06fddd58ed78-kube-api-access-gqdp8\") pod \"fc913c19-cac2-4d4e-93fb-06fddd58ed78\" (UID: \"fc913c19-cac2-4d4e-93fb-06fddd58ed78\") "
Oct 30 22:03:59.091722 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:59.078327    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc913c19-cac2-4d4e-93fb-06fddd58ed78-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc913c19-cac2-4d4e-93fb-06fddd58ed78-workload-certs\") pod \"fc913c19-cac2-4d4e-93fb-06fddd58ed78\" (UID: \"fc913c19-cac2-4d4e-93fb-06fddd58ed78\") "
Oct 30 22:03:59.091722 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:59.078377    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc913c19-cac2-4d4e-93fb-06fddd58ed78-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc913c19-cac2-4d4e-93fb-06fddd58ed78-istiod-ca-cert\") pod \"fc913c19-cac2-4d4e-93fb-06fddd58ed78\" (UID: \"fc913c19-cac2-4d4e-93fb-06fddd58ed78\") "
Oct 30 22:03:59.091722 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:59.078472    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fc913c19-cac2-4d4e-93fb-06fddd58ed78-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc913c19-cac2-4d4e-93fb-06fddd58ed78-istio-podinfo\") pod \"fc913c19-cac2-4d4e-93fb-06fddd58ed78\" (UID: \"fc913c19-cac2-4d4e-93fb-06fddd58ed78\") "
Oct 30 22:03:59.091722 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:59.078519    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc913c19-cac2-4d4e-93fb-06fddd58ed78-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc913c19-cac2-4d4e-93fb-06fddd58ed78-istio-data\") pod \"fc913c19-cac2-4d4e-93fb-06fddd58ed78\" (UID: \"fc913c19-cac2-4d4e-93fb-06fddd58ed78\") "
Oct 30 22:03:59.091722 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:59.078668    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc913c19-cac2-4d4e-93fb-06fddd58ed78-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc913c19-cac2-4d4e-93fb-06fddd58ed78-workload-socket\") pod \"fc913c19-cac2-4d4e-93fb-06fddd58ed78\" (UID: \"fc913c19-cac2-4d4e-93fb-06fddd58ed78\") "
Oct 30 22:03:59.091722 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:59.078781    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc913c19-cac2-4d4e-93fb-06fddd58ed78-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc913c19-cac2-4d4e-93fb-06fddd58ed78-credential-socket\") pod \"fc913c19-cac2-4d4e-93fb-06fddd58ed78\" (UID: \"fc913c19-cac2-4d4e-93fb-06fddd58ed78\") "
Oct 30 22:03:59.091722 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:59.078886    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc913c19-cac2-4d4e-93fb-06fddd58ed78-istio-token\" (UniqueName: \"kubernetes.io/projected/fc913c19-cac2-4d4e-93fb-06fddd58ed78-istio-token\") pod \"fc913c19-cac2-4d4e-93fb-06fddd58ed78\" (UID: \"fc913c19-cac2-4d4e-93fb-06fddd58ed78\") "
Oct 30 22:03:59.091722 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:59.084113    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc913c19-cac2-4d4e-93fb-06fddd58ed78-istio-token" pod "fc913c19-cac2-4d4e-93fb-06fddd58ed78" (UID: "fc913c19-cac2-4d4e-93fb-06fddd58ed78"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:03:59.091722 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:59.086662    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc913c19-cac2-4d4e-93fb-06fddd58ed78-istio-envoy" pod "fc913c19-cac2-4d4e-93fb-06fddd58ed78" (UID: "fc913c19-cac2-4d4e-93fb-06fddd58ed78"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:03:59.091722 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:59.088840    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc913c19-cac2-4d4e-93fb-06fddd58ed78-kube-api-access-gqdp8" pod "fc913c19-cac2-4d4e-93fb-06fddd58ed78" (UID: "fc913c19-cac2-4d4e-93fb-06fddd58ed78"). InnerVolumeSpecName "kube-api-access-gqdp8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:03:59.091722 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:59.089044    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc913c19-cac2-4d4e-93fb-06fddd58ed78-workload-certs" pod "fc913c19-cac2-4d4e-93fb-06fddd58ed78" (UID: "fc913c19-cac2-4d4e-93fb-06fddd58ed78"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:03:59.091722 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:59.089305    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc913c19-cac2-4d4e-93fb-06fddd58ed78-istiod-ca-cert" pod "fc913c19-cac2-4d4e-93fb-06fddd58ed78" (UID: "fc913c19-cac2-4d4e-93fb-06fddd58ed78"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:03:59.091722 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:59.090968    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc913c19-cac2-4d4e-93fb-06fddd58ed78-workload-socket" pod "fc913c19-cac2-4d4e-93fb-06fddd58ed78" (UID: "fc913c19-cac2-4d4e-93fb-06fddd58ed78"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:03:59.091722 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:59.091509    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fc913c19-cac2-4d4e-93fb-06fddd58ed78-istio-podinfo" pod "fc913c19-cac2-4d4e-93fb-06fddd58ed78" (UID: "fc913c19-cac2-4d4e-93fb-06fddd58ed78"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:03:59.093426 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:59.093400    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc913c19-cac2-4d4e-93fb-06fddd58ed78-credential-socket" pod "fc913c19-cac2-4d4e-93fb-06fddd58ed78" (UID: "fc913c19-cac2-4d4e-93fb-06fddd58ed78"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:03:59.093577 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:59.093425    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc913c19-cac2-4d4e-93fb-06fddd58ed78-istio-data" pod "fc913c19-cac2-4d4e-93fb-06fddd58ed78" (UID: "fc913c19-cac2-4d4e-93fb-06fddd58ed78"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:03:59.163561 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:59.160739    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-znswb"
Oct 30 22:03:59.185553 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:59.183846    5061 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc913c19-cac2-4d4e-93fb-06fddd58ed78-istio-podinfo\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:59.185553 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:59.183883    5061 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc913c19-cac2-4d4e-93fb-06fddd58ed78-istio-data\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:59.185553 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:59.183899    5061 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc913c19-cac2-4d4e-93fb-06fddd58ed78-workload-socket\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:59.185553 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:59.183915    5061 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc913c19-cac2-4d4e-93fb-06fddd58ed78-credential-socket\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:59.185553 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:59.183929    5061 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc913c19-cac2-4d4e-93fb-06fddd58ed78-istio-token\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:59.185553 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:59.183942    5061 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc913c19-cac2-4d4e-93fb-06fddd58ed78-istio-envoy\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:59.185553 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:59.183957    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqdp8\" (UniqueName: \"kubernetes.io/projected/fc913c19-cac2-4d4e-93fb-06fddd58ed78-kube-api-access-gqdp8\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:59.185553 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:59.183972    5061 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc913c19-cac2-4d4e-93fb-06fddd58ed78-workload-certs\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:59.185553 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:59.183985    5061 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc913c19-cac2-4d4e-93fb-06fddd58ed78-istiod-ca-cert\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:59.394732 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:59.387716    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8cf2cfc-250e-47a7-a8ef-25f6e343afd8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8cf2cfc-250e-47a7-a8ef-25f6e343afd8-workload-certs\") pod \"e8cf2cfc-250e-47a7-a8ef-25f6e343afd8\" (UID: \"e8cf2cfc-250e-47a7-a8ef-25f6e343afd8\") "
Oct 30 22:03:59.394732 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:59.387764    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8cf2cfc-250e-47a7-a8ef-25f6e343afd8-kube-api-access-s8k7f\" (UniqueName: \"kubernetes.io/projected/e8cf2cfc-250e-47a7-a8ef-25f6e343afd8-kube-api-access-s8k7f\") pod \"e8cf2cfc-250e-47a7-a8ef-25f6e343afd8\" (UID: \"e8cf2cfc-250e-47a7-a8ef-25f6e343afd8\") "
Oct 30 22:03:59.394732 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:59.388095    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8cf2cfc-250e-47a7-a8ef-25f6e343afd8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8cf2cfc-250e-47a7-a8ef-25f6e343afd8-istiod-ca-cert\") pod \"e8cf2cfc-250e-47a7-a8ef-25f6e343afd8\" (UID: \"e8cf2cfc-250e-47a7-a8ef-25f6e343afd8\") "
Oct 30 22:03:59.394732 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:59.388143    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8cf2cfc-250e-47a7-a8ef-25f6e343afd8-istio-token\" (UniqueName: \"kubernetes.io/projected/e8cf2cfc-250e-47a7-a8ef-25f6e343afd8-istio-token\") pod \"e8cf2cfc-250e-47a7-a8ef-25f6e343afd8\" (UID: \"e8cf2cfc-250e-47a7-a8ef-25f6e343afd8\") "
Oct 30 22:03:59.394732 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:59.388174    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8cf2cfc-250e-47a7-a8ef-25f6e343afd8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8cf2cfc-250e-47a7-a8ef-25f6e343afd8-credential-socket\") pod \"e8cf2cfc-250e-47a7-a8ef-25f6e343afd8\" (UID: \"e8cf2cfc-250e-47a7-a8ef-25f6e343afd8\") "
Oct 30 22:03:59.394732 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:59.388314    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e8cf2cfc-250e-47a7-a8ef-25f6e343afd8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8cf2cfc-250e-47a7-a8ef-25f6e343afd8-istio-podinfo\") pod \"e8cf2cfc-250e-47a7-a8ef-25f6e343afd8\" (UID: \"e8cf2cfc-250e-47a7-a8ef-25f6e343afd8\") "
Oct 30 22:03:59.394732 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:59.388425    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8cf2cfc-250e-47a7-a8ef-25f6e343afd8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8cf2cfc-250e-47a7-a8ef-25f6e343afd8-workload-socket\") pod \"e8cf2cfc-250e-47a7-a8ef-25f6e343afd8\" (UID: \"e8cf2cfc-250e-47a7-a8ef-25f6e343afd8\") "
Oct 30 22:03:59.394732 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:59.388538    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8cf2cfc-250e-47a7-a8ef-25f6e343afd8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8cf2cfc-250e-47a7-a8ef-25f6e343afd8-istio-envoy\") pod \"e8cf2cfc-250e-47a7-a8ef-25f6e343afd8\" (UID: \"e8cf2cfc-250e-47a7-a8ef-25f6e343afd8\") "
Oct 30 22:03:59.394732 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:59.388650    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8cf2cfc-250e-47a7-a8ef-25f6e343afd8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8cf2cfc-250e-47a7-a8ef-25f6e343afd8-istio-data\") pod \"e8cf2cfc-250e-47a7-a8ef-25f6e343afd8\" (UID: \"e8cf2cfc-250e-47a7-a8ef-25f6e343afd8\") "
Oct 30 22:03:59.394732 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:59.390870    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8cf2cfc-250e-47a7-a8ef-25f6e343afd8-istiod-ca-cert" pod "e8cf2cfc-250e-47a7-a8ef-25f6e343afd8" (UID: "e8cf2cfc-250e-47a7-a8ef-25f6e343afd8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:03:59.394732 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:59.391152    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8cf2cfc-250e-47a7-a8ef-25f6e343afd8-workload-certs" pod "e8cf2cfc-250e-47a7-a8ef-25f6e343afd8" (UID: "e8cf2cfc-250e-47a7-a8ef-25f6e343afd8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:03:59.394732 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:59.391732    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8cf2cfc-250e-47a7-a8ef-25f6e343afd8-workload-socket" pod "e8cf2cfc-250e-47a7-a8ef-25f6e343afd8" (UID: "e8cf2cfc-250e-47a7-a8ef-25f6e343afd8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:03:59.395892 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:59.394917    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8cf2cfc-250e-47a7-a8ef-25f6e343afd8-istio-token" pod "e8cf2cfc-250e-47a7-a8ef-25f6e343afd8" (UID: "e8cf2cfc-250e-47a7-a8ef-25f6e343afd8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:03:59.395892 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:59.395236    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8cf2cfc-250e-47a7-a8ef-25f6e343afd8-kube-api-access-s8k7f" pod "e8cf2cfc-250e-47a7-a8ef-25f6e343afd8" (UID: "e8cf2cfc-250e-47a7-a8ef-25f6e343afd8"). InnerVolumeSpecName "kube-api-access-s8k7f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:03:59.396125 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:59.396091    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8cf2cfc-250e-47a7-a8ef-25f6e343afd8-istio-data" pod "e8cf2cfc-250e-47a7-a8ef-25f6e343afd8" (UID: "e8cf2cfc-250e-47a7-a8ef-25f6e343afd8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:03:59.396619 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:59.396596    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8cf2cfc-250e-47a7-a8ef-25f6e343afd8-istio-envoy" pod "e8cf2cfc-250e-47a7-a8ef-25f6e343afd8" (UID: "e8cf2cfc-250e-47a7-a8ef-25f6e343afd8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:03:59.398430 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:59.398395    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e8cf2cfc-250e-47a7-a8ef-25f6e343afd8-istio-podinfo" pod "e8cf2cfc-250e-47a7-a8ef-25f6e343afd8" (UID: "e8cf2cfc-250e-47a7-a8ef-25f6e343afd8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:03:59.416704 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:59.416648    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8cf2cfc-250e-47a7-a8ef-25f6e343afd8-credential-socket" pod "e8cf2cfc-250e-47a7-a8ef-25f6e343afd8" (UID: "e8cf2cfc-250e-47a7-a8ef-25f6e343afd8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:03:59.494328 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:59.483978    5061 patch_prober.go:29] interesting pod/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-kmtkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Oct 30 22:03:59.494328 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:59.484032    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-kmtkh" podUID="03533dfe-8974-487d-9af6-4adda6fc8d8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Oct 30 22:03:59.494328 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:59.490711    5061 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8cf2cfc-250e-47a7-a8ef-25f6e343afd8-istiod-ca-cert\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:59.494328 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:59.490739    5061 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8cf2cfc-250e-47a7-a8ef-25f6e343afd8-istio-token\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:59.494328 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:59.490777    5061 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8cf2cfc-250e-47a7-a8ef-25f6e343afd8-credential-socket\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:59.494328 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:59.490791    5061 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8cf2cfc-250e-47a7-a8ef-25f6e343afd8-istio-podinfo\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:59.494328 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:59.490807    5061 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8cf2cfc-250e-47a7-a8ef-25f6e343afd8-workload-socket\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:59.494328 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:59.490832    5061 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8cf2cfc-250e-47a7-a8ef-25f6e343afd8-istio-envoy\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:59.494328 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:59.490847    5061 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8cf2cfc-250e-47a7-a8ef-25f6e343afd8-istio-data\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:59.494328 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:59.490861    5061 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8cf2cfc-250e-47a7-a8ef-25f6e343afd8-workload-certs\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:59.494328 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:59.490874    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8k7f\" (UniqueName: \"kubernetes.io/projected/e8cf2cfc-250e-47a7-a8ef-25f6e343afd8-kube-api-access-s8k7f\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:59.498054 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:59.496782    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4fx5l"
Oct 30 22:03:59.593162 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:59.592546    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c55fb717-8e3a-4c50-b0e7-249115db4dda-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c55fb717-8e3a-4c50-b0e7-249115db4dda-credential-socket\") pod \"c55fb717-8e3a-4c50-b0e7-249115db4dda\" (UID: \"c55fb717-8e3a-4c50-b0e7-249115db4dda\") "
Oct 30 22:03:59.593162 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:59.592606    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c55fb717-8e3a-4c50-b0e7-249115db4dda-istio-token\" (UniqueName: \"kubernetes.io/projected/c55fb717-8e3a-4c50-b0e7-249115db4dda-istio-token\") pod \"c55fb717-8e3a-4c50-b0e7-249115db4dda\" (UID: \"c55fb717-8e3a-4c50-b0e7-249115db4dda\") "
Oct 30 22:03:59.593162 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:59.592679    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c55fb717-8e3a-4c50-b0e7-249115db4dda-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c55fb717-8e3a-4c50-b0e7-249115db4dda-workload-socket\") pod \"c55fb717-8e3a-4c50-b0e7-249115db4dda\" (UID: \"c55fb717-8e3a-4c50-b0e7-249115db4dda\") "
Oct 30 22:03:59.593162 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:59.592733    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c55fb717-8e3a-4c50-b0e7-249115db4dda-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c55fb717-8e3a-4c50-b0e7-249115db4dda-istio-podinfo\") pod \"c55fb717-8e3a-4c50-b0e7-249115db4dda\" (UID: \"c55fb717-8e3a-4c50-b0e7-249115db4dda\") "
Oct 30 22:03:59.593162 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:59.592821    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c55fb717-8e3a-4c50-b0e7-249115db4dda-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c55fb717-8e3a-4c50-b0e7-249115db4dda-workload-certs\") pod \"c55fb717-8e3a-4c50-b0e7-249115db4dda\" (UID: \"c55fb717-8e3a-4c50-b0e7-249115db4dda\") "
Oct 30 22:03:59.593162 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:59.592903    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c55fb717-8e3a-4c50-b0e7-249115db4dda-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c55fb717-8e3a-4c50-b0e7-249115db4dda-istio-data\") pod \"c55fb717-8e3a-4c50-b0e7-249115db4dda\" (UID: \"c55fb717-8e3a-4c50-b0e7-249115db4dda\") "
Oct 30 22:03:59.593162 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:59.593004    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c55fb717-8e3a-4c50-b0e7-249115db4dda-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c55fb717-8e3a-4c50-b0e7-249115db4dda-istiod-ca-cert\") pod \"c55fb717-8e3a-4c50-b0e7-249115db4dda\" (UID: \"c55fb717-8e3a-4c50-b0e7-249115db4dda\") "
Oct 30 22:03:59.593162 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:59.593079    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c55fb717-8e3a-4c50-b0e7-249115db4dda-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c55fb717-8e3a-4c50-b0e7-249115db4dda-istio-envoy\") pod \"c55fb717-8e3a-4c50-b0e7-249115db4dda\" (UID: \"c55fb717-8e3a-4c50-b0e7-249115db4dda\") "
Oct 30 22:03:59.595429 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:59.593822    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c55fb717-8e3a-4c50-b0e7-249115db4dda-credential-socket" pod "c55fb717-8e3a-4c50-b0e7-249115db4dda" (UID: "c55fb717-8e3a-4c50-b0e7-249115db4dda"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:03:59.595429 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:59.594036    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c55fb717-8e3a-4c50-b0e7-249115db4dda-istio-data" pod "c55fb717-8e3a-4c50-b0e7-249115db4dda" (UID: "c55fb717-8e3a-4c50-b0e7-249115db4dda"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:03:59.595429 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:59.594231    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c55fb717-8e3a-4c50-b0e7-249115db4dda-workload-certs" pod "c55fb717-8e3a-4c50-b0e7-249115db4dda" (UID: "c55fb717-8e3a-4c50-b0e7-249115db4dda"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:03:59.595429 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:59.594414    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c55fb717-8e3a-4c50-b0e7-249115db4dda-workload-socket" pod "c55fb717-8e3a-4c50-b0e7-249115db4dda" (UID: "c55fb717-8e3a-4c50-b0e7-249115db4dda"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:03:59.595429 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:59.594738    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c55fb717-8e3a-4c50-b0e7-249115db4dda-istiod-ca-cert" pod "c55fb717-8e3a-4c50-b0e7-249115db4dda" (UID: "c55fb717-8e3a-4c50-b0e7-249115db4dda"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:03:59.596160 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:59.595823    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c55fb717-8e3a-4c50-b0e7-249115db4dda-kube-api-access-nf6cj\" (UniqueName: \"kubernetes.io/projected/c55fb717-8e3a-4c50-b0e7-249115db4dda-kube-api-access-nf6cj\") pod \"c55fb717-8e3a-4c50-b0e7-249115db4dda\" (UID: \"c55fb717-8e3a-4c50-b0e7-249115db4dda\") "
Oct 30 22:03:59.597186 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:59.597160    5061 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c55fb717-8e3a-4c50-b0e7-249115db4dda-istio-data\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:59.597312 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:59.597295    5061 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c55fb717-8e3a-4c50-b0e7-249115db4dda-istiod-ca-cert\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:59.597420 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:59.597405    5061 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c55fb717-8e3a-4c50-b0e7-249115db4dda-credential-socket\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:59.597665 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:59.597510    5061 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c55fb717-8e3a-4c50-b0e7-249115db4dda-workload-socket\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:59.600043 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:59.597764    5061 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c55fb717-8e3a-4c50-b0e7-249115db4dda-workload-certs\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:59.605502 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:59.605480    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c55fb717-8e3a-4c50-b0e7-249115db4dda-kube-api-access-nf6cj" pod "c55fb717-8e3a-4c50-b0e7-249115db4dda" (UID: "c55fb717-8e3a-4c50-b0e7-249115db4dda"). InnerVolumeSpecName "kube-api-access-nf6cj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:03:59.606017 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:59.605981    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c55fb717-8e3a-4c50-b0e7-249115db4dda-istio-envoy" pod "c55fb717-8e3a-4c50-b0e7-249115db4dda" (UID: "c55fb717-8e3a-4c50-b0e7-249115db4dda"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:03:59.700139 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:59.700021    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nf6cj\" (UniqueName: \"kubernetes.io/projected/c55fb717-8e3a-4c50-b0e7-249115db4dda-kube-api-access-nf6cj\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:59.700348 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:59.700330    5061 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c55fb717-8e3a-4c50-b0e7-249115db4dda-istio-envoy\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:59.777860 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:59.777817    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c55fb717-8e3a-4c50-b0e7-249115db4dda-istio-podinfo" pod "c55fb717-8e3a-4c50-b0e7-249115db4dda" (UID: "c55fb717-8e3a-4c50-b0e7-249115db4dda"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:03:59.779622 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:59.779592    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c55fb717-8e3a-4c50-b0e7-249115db4dda-istio-token" pod "c55fb717-8e3a-4c50-b0e7-249115db4dda" (UID: "c55fb717-8e3a-4c50-b0e7-249115db4dda"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:03:59.802597 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:59.801710    5061 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c55fb717-8e3a-4c50-b0e7-249115db4dda-istio-token\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:59.802597 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:59.801743    5061 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c55fb717-8e3a-4c50-b0e7-249115db4dda-istio-podinfo\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:59.869359 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:59.869268    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-znswb" event={"ID":"e8cf2cfc-250e-47a7-a8ef-25f6e343afd8","Type":"ContainerDied","Data":"bac251b3c902469a0f1c8fa914a4fb2a11edd9c35489095413aeec6589e83237"}
Oct 30 22:03:59.869611 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:59.869412    5061 scope.go:122] "RemoveContainer" containerID="409f68272498f576ec2ea40a91a3f64d337a7cee24bcd68b568911882de9d529"
Oct 30 22:03:59.869711 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:59.869681    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-znswb"
Oct 30 22:03:59.880494 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:59.880465    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4fx5l" event={"ID":"c55fb717-8e3a-4c50-b0e7-249115db4dda","Type":"ContainerDied","Data":"202f1e6257937f0892d23c56faf4f05726c55ece844f181d9c6b7c1d88440e24"}
Oct 30 22:03:59.880920 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:59.880891    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4fx5l"
Oct 30 22:03:59.888861 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:59.888834    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gg4hd"
Oct 30 22:03:59.888972 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:59.888880    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gg4hd" event={"ID":"fc913c19-cac2-4d4e-93fb-06fddd58ed78","Type":"ContainerDied","Data":"c5bb69d6a6c633c9dba4e801610dc417e891df369010884dca2bafee63f40523"}
Oct 30 22:03:59.896518 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:59.895998    5061 scope.go:122] "RemoveContainer" containerID="ff20afd34720205e27cecdf86435ddb1ba57242f30d0310531184f407f2a3fab"
Oct 30 22:03:59.963605 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:03:59.961676    5061 scope.go:122] "RemoveContainer" containerID="866afe08bd97e1bcfcdf42a15293ecc14d98de6c1951e2a2a30d7fe24a97bccb"
Oct 30 22:04:00.086745 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:00.078782    5061 patch_prober.go:29] interesting pod/e2e-worker-w3-f776cb-openshift-default-74d577586f-5z645 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Oct 30 22:04:00.086745 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:00.078839    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-5z645" podUID="a84e4890-ba61-49a5-b9b6-be9cf741d2d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Oct 30 22:04:00.087209 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:00.087185    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-znswb"]
Oct 30 22:04:00.090182 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:00.090156    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-znswb"]
Oct 30 22:04:00.294615 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:00.290598    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gg4hd"]
Oct 30 22:04:00.305562 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:00.305522    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-gg4hd"]
Oct 30 22:04:00.376021 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:00.375229    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4fx5l"]
Oct 30 22:04:00.382043 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:00.382017    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4fx5l"]
Oct 30 22:04:00.508579 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:00.475651    5061 patch_prober.go:29] interesting pod/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-kmtkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Oct 30 22:04:00.508579 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:00.475728    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-kmtkh" podUID="03533dfe-8974-487d-9af6-4adda6fc8d8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Oct 30 22:04:00.508579 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:00.484877    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27350177-mdzkw"]
Oct 30 22:04:00.508579 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:00.486467    5061 cpu_manager.go:414] "containerMap: removing container" podUID="c55fb717-8e3a-4c50-b0e7-249115db4dda" containerName="istio-proxy"
Oct 30 22:04:00.508579 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:00.486493    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c55fb717-8e3a-4c50-b0e7-249115db4dda" containerName="istio-proxy"
Oct 30 22:04:00.508579 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:00.486511    5061 cpu_manager.go:414] "containerMap: removing container" podUID="e8cf2cfc-250e-47a7-a8ef-25f6e343afd8" containerName="istio-proxy"
Oct 30 22:04:00.508579 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:00.486519    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8cf2cfc-250e-47a7-a8ef-25f6e343afd8" containerName="istio-proxy"
Oct 30 22:04:00.508579 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:00.486571    5061 cpu_manager.go:414] "containerMap: removing container" podUID="fc913c19-cac2-4d4e-93fb-06fddd58ed78" containerName="istio-proxy"
Oct 30 22:04:00.508579 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:00.486578    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc913c19-cac2-4d4e-93fb-06fddd58ed78" containerName="istio-proxy"
Oct 30 22:04:00.508579 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:00.487000    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8cf2cfc-250e-47a7-a8ef-25f6e343afd8" containerName="istio-proxy"
Oct 30 22:04:00.508579 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:00.487025    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="c55fb717-8e3a-4c50-b0e7-249115db4dda" containerName="istio-proxy"
Oct 30 22:04:00.508579 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:00.487039    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc913c19-cac2-4d4e-93fb-06fddd58ed78" containerName="istio-proxy"
Oct 30 22:04:00.508579 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:00.501871    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27350177-mdzkw"
Oct 30 22:04:00.596576 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:00.596066    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:04:00.596576 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:00.596205    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gdmfg\""
Oct 30 22:04:00.597848 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:00.597570    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjqv5\" (UniqueName: \"kubernetes.io/projected/c9665869-1cbf-484c-89e1-d25f21b7ec54-kube-api-access-cjqv5\") pod \"maas-api-revoke-sub-27350177-mdzkw\" (UID: \"c9665869-1cbf-484c-89e1-d25f21b7ec54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27350177-mdzkw"
Oct 30 22:04:00.597848 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:00.597777    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9665869-1cbf-484c-89e1-d25f21b7ec54-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-27350177-mdzkw\" (UID: \"c9665869-1cbf-484c-89e1-d25f21b7ec54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27350177-mdzkw"
Oct 30 22:04:00.597848 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:00.597819    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9665869-1cbf-484c-89e1-d25f21b7ec54-maas-api-xks-ca\") pod \"maas-api-revoke-sub-27350177-mdzkw\" (UID: \"c9665869-1cbf-484c-89e1-d25f21b7ec54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27350177-mdzkw"
Oct 30 22:04:00.603023 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:00.602996    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c55fb717-8e3a-4c50-b0e7-249115db4dda" path="/var/lib/kubelet/pods/c55fb717-8e3a-4c50-b0e7-249115db4dda/volumes"
Oct 30 22:04:00.603633 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:00.603610    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8cf2cfc-250e-47a7-a8ef-25f6e343afd8" path="/var/lib/kubelet/pods/e8cf2cfc-250e-47a7-a8ef-25f6e343afd8/volumes"
Oct 30 22:04:00.604156 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:00.604136    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc913c19-cac2-4d4e-93fb-06fddd58ed78" path="/var/lib/kubelet/pods/fc913c19-cac2-4d4e-93fb-06fddd58ed78/volumes"
Oct 30 22:04:00.604620 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:00.604568    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27350177-mdzkw"]
Oct 30 22:04:00.703388 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:00.703354    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9665869-1cbf-484c-89e1-d25f21b7ec54-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-27350177-mdzkw\" (UID: \"c9665869-1cbf-484c-89e1-d25f21b7ec54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27350177-mdzkw"
Oct 30 22:04:00.703672 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:00.703646    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9665869-1cbf-484c-89e1-d25f21b7ec54-maas-api-xks-ca\") pod \"maas-api-revoke-sub-27350177-mdzkw\" (UID: \"c9665869-1cbf-484c-89e1-d25f21b7ec54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27350177-mdzkw"
Oct 30 22:04:00.703992 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:00.703970    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjqv5\" (UniqueName: \"kubernetes.io/projected/c9665869-1cbf-484c-89e1-d25f21b7ec54-kube-api-access-cjqv5\") pod \"maas-api-revoke-sub-27350177-mdzkw\" (UID: \"c9665869-1cbf-484c-89e1-d25f21b7ec54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27350177-mdzkw"
Oct 30 22:04:00.704289 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:00.704260    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9665869-1cbf-484c-89e1-d25f21b7ec54-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-27350177-mdzkw\" (UID: \"c9665869-1cbf-484c-89e1-d25f21b7ec54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27350177-mdzkw"
Oct 30 22:04:00.707521 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:00.707443    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9665869-1cbf-484c-89e1-d25f21b7ec54-maas-api-xks-ca\") pod \"maas-api-revoke-sub-27350177-mdzkw\" (UID: \"c9665869-1cbf-484c-89e1-d25f21b7ec54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27350177-mdzkw"
Oct 30 22:04:00.784784 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:00.784751    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjqv5\" (UniqueName: \"kubernetes.io/projected/c9665869-1cbf-484c-89e1-d25f21b7ec54-kube-api-access-cjqv5\") pod \"maas-api-revoke-sub-27350177-mdzkw\" (UID: \"c9665869-1cbf-484c-89e1-d25f21b7ec54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27350177-mdzkw"
Oct 30 22:04:00.905064 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:00.904983    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27350177-mdzkw"
Oct 30 22:04:00.976476 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:00.976442    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8c7d3fc-qjfvv"]
Oct 30 22:04:00.987074 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:00.987039    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8c7d3fc-qjfvv"
Oct 30 22:04:01.078547 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:01.076158    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7780bf6d-2d91-4d60-b073-6cc0b1b7009e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d8c7d3fc-qjfvv\" (UID: \"7780bf6d-2d91-4d60-b073-6cc0b1b7009e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8c7d3fc-qjfvv"
Oct 30 22:04:01.078547 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:01.076230    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7780bf6d-2d91-4d60-b073-6cc0b1b7009e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d8c7d3fc-qjfvv\" (UID: \"7780bf6d-2d91-4d60-b073-6cc0b1b7009e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8c7d3fc-qjfvv"
Oct 30 22:04:01.078547 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:01.076313    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmp6r\" (UniqueName: \"kubernetes.io/projected/7780bf6d-2d91-4d60-b073-6cc0b1b7009e-kube-api-access-kmp6r\") pod \"maas-api-revoke-sub-d8c7d3fc-qjfvv\" (UID: \"7780bf6d-2d91-4d60-b073-6cc0b1b7009e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8c7d3fc-qjfvv"
Oct 30 22:04:01.078547 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:01.077237    5061 patch_prober.go:29] interesting pod/e2e-worker-w3-f776cb-openshift-default-74d577586f-5z645 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Oct 30 22:04:01.078547 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:01.077288    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-5z645" podUID="a84e4890-ba61-49a5-b9b6-be9cf741d2d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Oct 30 22:04:01.102742 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:01.102424    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8c7d3fc-qjfvv"]
Oct 30 22:04:01.184924 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:01.184630    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmp6r\" (UniqueName: \"kubernetes.io/projected/7780bf6d-2d91-4d60-b073-6cc0b1b7009e-kube-api-access-kmp6r\") pod \"maas-api-revoke-sub-d8c7d3fc-qjfvv\" (UID: \"7780bf6d-2d91-4d60-b073-6cc0b1b7009e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8c7d3fc-qjfvv"
Oct 30 22:04:01.185032 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:01.184943    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7780bf6d-2d91-4d60-b073-6cc0b1b7009e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d8c7d3fc-qjfvv\" (UID: \"7780bf6d-2d91-4d60-b073-6cc0b1b7009e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8c7d3fc-qjfvv"
Oct 30 22:04:01.185032 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:01.185008    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7780bf6d-2d91-4d60-b073-6cc0b1b7009e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d8c7d3fc-qjfvv\" (UID: \"7780bf6d-2d91-4d60-b073-6cc0b1b7009e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8c7d3fc-qjfvv"
Oct 30 22:04:01.279587 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:01.193521    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7780bf6d-2d91-4d60-b073-6cc0b1b7009e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d8c7d3fc-qjfvv\" (UID: \"7780bf6d-2d91-4d60-b073-6cc0b1b7009e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8c7d3fc-qjfvv"
Oct 30 22:04:01.288071 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:01.286067    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7780bf6d-2d91-4d60-b073-6cc0b1b7009e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d8c7d3fc-qjfvv\" (UID: \"7780bf6d-2d91-4d60-b073-6cc0b1b7009e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8c7d3fc-qjfvv"
Oct 30 22:04:01.408578 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:01.405570    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmp6r\" (UniqueName: \"kubernetes.io/projected/7780bf6d-2d91-4d60-b073-6cc0b1b7009e-kube-api-access-kmp6r\") pod \"maas-api-revoke-sub-d8c7d3fc-qjfvv\" (UID: \"7780bf6d-2d91-4d60-b073-6cc0b1b7009e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8c7d3fc-qjfvv"
Oct 30 22:04:01.473601 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:01.472710    5061 patch_prober.go:29] interesting pod/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-kmtkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Oct 30 22:04:01.473601 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:01.472794    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-kmtkh" podUID="03533dfe-8974-487d-9af6-4adda6fc8d8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Oct 30 22:04:01.500192 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:01.500165    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8c7d3fc-qjfvv"
Oct 30 22:04:01.683242 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:01.683131    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27350177-mdzkw"]
Oct 30 22:04:01.997859 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:01.997563    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27350177-mdzkw" event={"ID":"c9665869-1cbf-484c-89e1-d25f21b7ec54","Type":"ContainerStarted","Data":"c487490a63f23474d45c36e81e5b7a049d2be87e235f0368a57c6d9416e10b83"}
Oct 30 22:04:02.042579 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:02.037773    5061 patch_prober.go:29] interesting pod/e2e-worker-w3-f776cb-openshift-default-74d577586f-5z645 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Oct 30 22:04:02.042579 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:02.037836    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-5z645" podUID="a84e4890-ba61-49a5-b9b6-be9cf741d2d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Oct 30 22:04:02.083925 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:02.083883    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8c7d3fc-qjfvv"]
Oct 30 22:04:02.145698 ip-10-0-130-60 kubenswrapper[5061]: W1030 22:04:02.144991    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7780bf6d_2d91_4d60_b073_6cc0b1b7009e.slice/crio-7722b96ed2cd621e80a3ceaa280133fdd6cb16d5354bfe41908f88850b92d5f8 WatchSource:0}: Error finding container 7722b96ed2cd621e80a3ceaa280133fdd6cb16d5354bfe41908f88850b92d5f8: Status 404 returned error can't find the container with id 7722b96ed2cd621e80a3ceaa280133fdd6cb16d5354bfe41908f88850b92d5f8
Oct 30 22:04:02.459421 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:02.459077    5061 patch_prober.go:29] interesting pod/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-kmtkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Oct 30 22:04:02.459421 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:02.459121    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-kmtkh" podUID="03533dfe-8974-487d-9af6-4adda6fc8d8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Oct 30 22:04:03.013265 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:03.013088    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8c7d3fc-qjfvv" event={"ID":"7780bf6d-2d91-4d60-b073-6cc0b1b7009e","Type":"ContainerStarted","Data":"b1be1536ac5dea3754f7823c3c90d337ec4430cb02fa9b31c65f93a1a1dfd0cf"}
Oct 30 22:04:03.013265 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:03.013161    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8c7d3fc-qjfvv" event={"ID":"7780bf6d-2d91-4d60-b073-6cc0b1b7009e","Type":"ContainerStarted","Data":"7722b96ed2cd621e80a3ceaa280133fdd6cb16d5354bfe41908f88850b92d5f8"}
Oct 30 22:04:03.018918 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:03.018880    5061 generic.go:356] "Generic (PLEG): container finished" podID="c9665869-1cbf-484c-89e1-d25f21b7ec54" containerID="d85ce38af7c029e19670695188b2d5eeb9c842425390a35b8e53692920adf996" exitCode=0
Oct 30 22:04:03.019073 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:03.018948    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27350177-mdzkw" event={"ID":"c9665869-1cbf-484c-89e1-d25f21b7ec54","Type":"ContainerDied","Data":"d85ce38af7c029e19670695188b2d5eeb9c842425390a35b8e53692920adf996"}
Oct 30 22:04:03.344904 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:03.344794    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8c7d3fc-qjfvv" podStartSLOduration=3.344774185 podStartE2EDuration="3.344774185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:03.140800352 +0000 UTC m=+1296.753570133" watchObservedRunningTime="2026-10-30 22:04:03.344774185 +0000 UTC m=+1296.957543949"
Oct 30 22:04:03.472748 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:03.470324    5061 patch_prober.go:29] interesting pod/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-kmtkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Oct 30 22:04:03.472748 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:03.470382    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-kmtkh" podUID="03533dfe-8974-487d-9af6-4adda6fc8d8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Oct 30 22:04:03.550767 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:03.549506    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-5z645"
Oct 30 22:04:03.551006 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:03.550937    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-5z645"
Oct 30 22:04:03.552232 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:03.552210    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-5z645"
Oct 30 22:04:04.038052 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:04.038015    5061 generic.go:356] "Generic (PLEG): container finished" podID="7780bf6d-2d91-4d60-b073-6cc0b1b7009e" containerID="b1be1536ac5dea3754f7823c3c90d337ec4430cb02fa9b31c65f93a1a1dfd0cf" exitCode=0
Oct 30 22:04:04.038553 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:04.038479    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8c7d3fc-qjfvv" event={"ID":"7780bf6d-2d91-4d60-b073-6cc0b1b7009e","Type":"ContainerDied","Data":"b1be1536ac5dea3754f7823c3c90d337ec4430cb02fa9b31c65f93a1a1dfd0cf"}
Oct 30 22:04:04.346560 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:04.345295    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27350177-mdzkw"
Oct 30 22:04:04.429643 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:04.428084    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9665869-1cbf-484c-89e1-d25f21b7ec54-kube-api-access-cjqv5\" (UniqueName: \"kubernetes.io/projected/c9665869-1cbf-484c-89e1-d25f21b7ec54-kube-api-access-cjqv5\") pod \"c9665869-1cbf-484c-89e1-d25f21b7ec54\" (UID: \"c9665869-1cbf-484c-89e1-d25f21b7ec54\") "
Oct 30 22:04:04.429643 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:04.428249    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c9665869-1cbf-484c-89e1-d25f21b7ec54-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9665869-1cbf-484c-89e1-d25f21b7ec54-maas-api-xks-ca\") pod \"c9665869-1cbf-484c-89e1-d25f21b7ec54\" (UID: \"c9665869-1cbf-484c-89e1-d25f21b7ec54\") "
Oct 30 22:04:04.429643 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:04.428379    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9665869-1cbf-484c-89e1-d25f21b7ec54-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9665869-1cbf-484c-89e1-d25f21b7ec54-maas-api-openshift-ca\") pod \"c9665869-1cbf-484c-89e1-d25f21b7ec54\" (UID: \"c9665869-1cbf-484c-89e1-d25f21b7ec54\") "
Oct 30 22:04:04.436591 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:04.434076    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9665869-1cbf-484c-89e1-d25f21b7ec54-maas-api-openshift-ca" pod "c9665869-1cbf-484c-89e1-d25f21b7ec54" (UID: "c9665869-1cbf-484c-89e1-d25f21b7ec54"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:04:04.456125 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:04.456091    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9665869-1cbf-484c-89e1-d25f21b7ec54-kube-api-access-cjqv5" pod "c9665869-1cbf-484c-89e1-d25f21b7ec54" (UID: "c9665869-1cbf-484c-89e1-d25f21b7ec54"). InnerVolumeSpecName "kube-api-access-cjqv5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:04:04.458937 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:04.458913    5061 patch_prober.go:29] interesting pod/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-kmtkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Oct 30 22:04:04.459519 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:04.459493    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-kmtkh" podUID="03533dfe-8974-487d-9af6-4adda6fc8d8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Oct 30 22:04:04.466997 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:04.459440    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9665869-1cbf-484c-89e1-d25f21b7ec54-maas-api-xks-ca" pod "c9665869-1cbf-484c-89e1-d25f21b7ec54" (UID: "c9665869-1cbf-484c-89e1-d25f21b7ec54"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:04:04.534037 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:04.531076    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9665869-1cbf-484c-89e1-d25f21b7ec54-maas-api-xks-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:04:04.534037 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:04.531115    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9665869-1cbf-484c-89e1-d25f21b7ec54-maas-api-openshift-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:04:04.534037 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:04.531130    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjqv5\" (UniqueName: \"kubernetes.io/projected/c9665869-1cbf-484c-89e1-d25f21b7ec54-kube-api-access-cjqv5\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:04:05.085083 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:05.085050    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27350177-mdzkw"
Oct 30 22:04:05.085262 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:05.085092    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27350177-mdzkw" event={"ID":"c9665869-1cbf-484c-89e1-d25f21b7ec54","Type":"ContainerDied","Data":"c487490a63f23474d45c36e81e5b7a049d2be87e235f0368a57c6d9416e10b83"}
Oct 30 22:04:05.085262 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:05.085130    5061 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c487490a63f23474d45c36e81e5b7a049d2be87e235f0368a57c6d9416e10b83"
Oct 30 22:04:05.338630 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:05.336327    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8c7d3fc-qjfvv"
Oct 30 22:04:05.351659 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:05.351629    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7780bf6d-2d91-4d60-b073-6cc0b1b7009e-kube-api-access-kmp6r\" (UniqueName: \"kubernetes.io/projected/7780bf6d-2d91-4d60-b073-6cc0b1b7009e-kube-api-access-kmp6r\") pod \"7780bf6d-2d91-4d60-b073-6cc0b1b7009e\" (UID: \"7780bf6d-2d91-4d60-b073-6cc0b1b7009e\") "
Oct 30 22:04:05.352090 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:05.351710    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7780bf6d-2d91-4d60-b073-6cc0b1b7009e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7780bf6d-2d91-4d60-b073-6cc0b1b7009e-maas-api-xks-ca\") pod \"7780bf6d-2d91-4d60-b073-6cc0b1b7009e\" (UID: \"7780bf6d-2d91-4d60-b073-6cc0b1b7009e\") "
Oct 30 22:04:05.352090 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:05.351838    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7780bf6d-2d91-4d60-b073-6cc0b1b7009e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7780bf6d-2d91-4d60-b073-6cc0b1b7009e-maas-api-openshift-ca\") pod \"7780bf6d-2d91-4d60-b073-6cc0b1b7009e\" (UID: \"7780bf6d-2d91-4d60-b073-6cc0b1b7009e\") "
Oct 30 22:04:05.352210 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:05.352185    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7780bf6d-2d91-4d60-b073-6cc0b1b7009e-maas-api-openshift-ca" pod "7780bf6d-2d91-4d60-b073-6cc0b1b7009e" (UID: "7780bf6d-2d91-4d60-b073-6cc0b1b7009e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:04:05.352693 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:05.352670    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7780bf6d-2d91-4d60-b073-6cc0b1b7009e-maas-api-openshift-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:04:05.353895 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:05.353868    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7780bf6d-2d91-4d60-b073-6cc0b1b7009e-kube-api-access-kmp6r" pod "7780bf6d-2d91-4d60-b073-6cc0b1b7009e" (UID: "7780bf6d-2d91-4d60-b073-6cc0b1b7009e"). InnerVolumeSpecName "kube-api-access-kmp6r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:04:05.354159 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:05.354133    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7780bf6d-2d91-4d60-b073-6cc0b1b7009e-maas-api-xks-ca" pod "7780bf6d-2d91-4d60-b073-6cc0b1b7009e" (UID: "7780bf6d-2d91-4d60-b073-6cc0b1b7009e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:04:05.454965 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:05.454925    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmp6r\" (UniqueName: \"kubernetes.io/projected/7780bf6d-2d91-4d60-b073-6cc0b1b7009e-kube-api-access-kmp6r\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:04:05.455186 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:05.455168    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7780bf6d-2d91-4d60-b073-6cc0b1b7009e-maas-api-xks-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:04:05.463312 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:05.463277    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-kmtkh"
Oct 30 22:04:05.463587 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:05.463565    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-kmtkh"
Oct 30 22:04:05.464837 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:05.464815    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-kmtkh"
Oct 30 22:04:06.094127 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:06.094095    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8c7d3fc-qjfvv"
Oct 30 22:04:06.094322 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:06.094129    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8c7d3fc-qjfvv" event={"ID":"7780bf6d-2d91-4d60-b073-6cc0b1b7009e","Type":"ContainerDied","Data":"7722b96ed2cd621e80a3ceaa280133fdd6cb16d5354bfe41908f88850b92d5f8"}
Oct 30 22:04:06.094322 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:06.094180    5061 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7722b96ed2cd621e80a3ceaa280133fdd6cb16d5354bfe41908f88850b92d5f8"
Oct 30 22:04:12.192553 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:12.183652    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2d1f91/premium-w0-2d1f91-kserve-6965b558cf-59lgw" podUID="fb84233d-25e0-48bc-8899-c3a3cb3cfeb9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:12.192553 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:12.188171    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5be0d1/premium-w2-5be0d1-kserve-759b6fb995-c6gp4" podUID="933f2470-68de-4768-a05b-6c454f16f5f2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded"
Oct 30 22:04:12.204052 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:12.198595    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-14af5c/premium-w6-14af5c-kserve-6549647f4-f4gwg" podUID="9b42ee01-c60e-462c-a0e0-b04a52014db6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:12.204052 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:12.198958    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-17d809/simulator-w4-17d809-kserve-78d67b6f4-7hsvb" podUID="e7337cf8-4f59-4afb-9b50-6b6c9fa4a773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:12.204052 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:12.199233    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a07620/simulator-w1-a07620-kserve-55b44674f4-4z9lk" podUID="efe7c063-fb0e-4679-8a7c-13f2a0abaa74" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Oct 30 22:04:12.279561 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:12.247402    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2d1f91/simulator-w0-2d1f91-kserve-7699765c74-jwtkb" podUID="44b7d6f6-32d8-4ef4-95bc-eef604fa6b5b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 22:04:12.279561 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:12.247830    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-17d809/premium-w4-17d809-kserve-7c9bdbbcbc-5mtmm" podUID="1e0eca81-7db1-482b-8dd6-5b28fb89e010" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Oct 30 22:04:12.279561 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:12.255345    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-14af5c/distinct-2-w6-14af5c-kserve-58d554bdb6-r5r6j" podUID="990f4e94-708e-45f3-b2ad-01cd50b3257b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:12.305551 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:12.303639    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5be0d1/distinct-w2-5be0d1-kserve-6b85698c8-sxn9k" podUID="7a2dbfc5-908f-41d1-baff-074c56d2cf16" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:12.350558 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:12.346633    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-17d809/distinct-w4-17d809-kserve-6b8f5c48cd-lhrwp" podUID="ef5e8691-42e1-471b-ba7d-90881028eca7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:12.699709 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:12.699663    5061 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-14af5c/distinct-2-w6-14af5c-kserve-58d554bdb6-r5r6j" podUID="990f4e94-708e-45f3-b2ad-01cd50b3257b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/health\": context deadline exceeded"
Oct 30 22:04:12.834704 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:12.834623    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-14af5c/distinct-w6-14af5c-kserve-7c7cb9d69c-hljl9" podUID="1f8d9486-fb2f-491a-a8d7-c23c21bd116a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:12.852416 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:12.851619    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/distinct-2-w3-f776cb-kserve-7b4d4cc954-d88pf" podUID="5b184062-7fac-43d4-ba5f-21567df85da3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:13.139553 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:13.126572    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-14af5c/simulator-w6-14af5c-kserve-85b8bd96d7-wlcb9" podUID="39766c7d-5f1d-42db-8920-c0321cd12c99" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Oct 30 22:04:13.190273 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:13.186058    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2d1f91/premium-w0-2d1f91-kserve-6965b558cf-59lgw" podUID="fb84233d-25e0-48bc-8899-c3a3cb3cfeb9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 30 22:04:13.190273 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:13.186430    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2d1f91/premium-w0-2d1f91-kserve-6965b558cf-59lgw"
Oct 30 22:04:13.314741 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:13.314697    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-blrrb" podUID="cb4b1ed5-2192-4877-8be7-ca82af831af5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": context deadline exceeded"
Oct 30 22:04:13.426727 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:13.326764    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-14af5c/premium-w6-14af5c-kserve-6549647f4-f4gwg" podUID="9b42ee01-c60e-462c-a0e0-b04a52014db6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded"
Oct 30 22:04:13.426727 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:13.326816    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a07620/simulator-w1-a07620-kserve-55b44674f4-4z9lk" podUID="efe7c063-fb0e-4679-8a7c-13f2a0abaa74" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Oct 30 22:04:13.426727 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:13.425425    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-14af5c/premium-w6-14af5c-kserve-6549647f4-f4gwg"
Oct 30 22:04:13.426727 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:13.425486    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a07620/simulator-w1-a07620-kserve-55b44674f4-4z9lk"
Oct 30 22:04:13.426727 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:13.328009    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2d1f91/simulator-w0-2d1f91-kserve-7699765c74-jwtkb" podUID="44b7d6f6-32d8-4ef4-95bc-eef604fa6b5b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 22:04:13.426727 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:13.328041    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-17d809/premium-w4-17d809-kserve-7c9bdbbcbc-5mtmm" podUID="1e0eca81-7db1-482b-8dd6-5b28fb89e010" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Oct 30 22:04:13.427417 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:13.344589    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5be0d1/premium-w2-5be0d1-kserve-759b6fb995-c6gp4" podUID="933f2470-68de-4768-a05b-6c454f16f5f2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded"
Oct 30 22:04:13.427553 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:13.345710    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-14af5c/distinct-2-w6-14af5c-kserve-58d554bdb6-r5r6j" podUID="990f4e94-708e-45f3-b2ad-01cd50b3257b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded"
Oct 30 22:04:13.447124 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:13.427747    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2d1f91/simulator-w0-2d1f91-kserve-7699765c74-jwtkb"
Oct 30 22:04:13.447124 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:13.427803    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-17d809/premium-w4-17d809-kserve-7c9bdbbcbc-5mtmm"
Oct 30 22:04:13.447124 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:13.427839    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-5be0d1/premium-w2-5be0d1-kserve-759b6fb995-c6gp4"
Oct 30 22:04:13.447124 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:13.346070    5061 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-14af5c/distinct-w6-14af5c-kserve-7c7cb9d69c-hljl9" podUID="1f8d9486-fb2f-491a-a8d7-c23c21bd116a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/health\": context deadline exceeded"
Oct 30 22:04:13.447124 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:13.397584    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-17d809/simulator-w4-17d809-kserve-78d67b6f4-7hsvb" podUID="e7337cf8-4f59-4afb-9b50-6b6c9fa4a773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:13.447124 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:13.346886    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5be0d1/distinct-w2-5be0d1-kserve-6b85698c8-sxn9k" podUID="7a2dbfc5-908f-41d1-baff-074c56d2cf16" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 30 22:04:13.447124 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:13.346933    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-17d809/distinct-w4-17d809-kserve-6b8f5c48cd-lhrwp" podUID="ef5e8691-42e1-471b-ba7d-90881028eca7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:13.447124 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:13.439770    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-17d809/simulator-w4-17d809-kserve-78d67b6f4-7hsvb"
Oct 30 22:04:13.447124 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:13.439824    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-14af5c/distinct-2-w6-14af5c-kserve-58d554bdb6-r5r6j"
Oct 30 22:04:13.447124 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:13.439865    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-5be0d1/distinct-w2-5be0d1-kserve-6b85698c8-sxn9k"
Oct 30 22:04:13.447124 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:13.439900    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-17d809/distinct-w4-17d809-kserve-6b8f5c48cd-lhrwp"
Oct 30 22:04:13.556119 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:13.556078    5061 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-14af5c/premium-w6-14af5c-kserve-6549647f4-f4gwg" podUID="9b42ee01-c60e-462c-a0e0-b04a52014db6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:13.877579 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:13.877465    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/distinct-2-w3-f776cb-kserve-7b4d4cc954-d88pf" podUID="5b184062-7fac-43d4-ba5f-21567df85da3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:13.913645 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:13.899217    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-14af5c/distinct-w6-14af5c-kserve-7c7cb9d69c-hljl9" podUID="1f8d9486-fb2f-491a-a8d7-c23c21bd116a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:13.927108 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:13.914091    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f776cb/distinct-2-w3-f776cb-kserve-7b4d4cc954-d88pf"
Oct 30 22:04:13.927485 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:13.927458    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-14af5c/distinct-w6-14af5c-kserve-7c7cb9d69c-hljl9"
Oct 30 22:04:14.123015 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:14.122970    5061 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-5be0d1/distinct-w2-5be0d1-kserve-6b85698c8-sxn9k" podUID="7a2dbfc5-908f-41d1-baff-074c56d2cf16" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": context deadline exceeded"
Oct 30 22:04:14.128792 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:14.128700    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-14af5c/simulator-w6-14af5c-kserve-85b8bd96d7-wlcb9" podUID="39766c7d-5f1d-42db-8920-c0321cd12c99" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:14.153723 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:14.153686    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-14af5c/simulator-w6-14af5c-kserve-85b8bd96d7-wlcb9"
Oct 30 22:04:14.210689 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:14.210638    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2d1f91/premium-w0-2d1f91-kserve-6965b558cf-59lgw" podUID="fb84233d-25e0-48bc-8899-c3a3cb3cfeb9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 30 22:04:14.503704 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:14.492226    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-blrrb" podUID="cb4b1ed5-2192-4877-8be7-ca82af831af5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": context deadline exceeded"
Oct 30 22:04:14.503704 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:14.492621    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-14af5c/premium-w6-14af5c-kserve-6549647f4-f4gwg" podUID="9b42ee01-c60e-462c-a0e0-b04a52014db6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded"
Oct 30 22:04:14.503704 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:14.492834    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a07620/simulator-w1-a07620-kserve-55b44674f4-4z9lk" podUID="efe7c063-fb0e-4679-8a7c-13f2a0abaa74" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Oct 30 22:04:14.503704 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:14.493074    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2d1f91/simulator-w0-2d1f91-kserve-7699765c74-jwtkb" podUID="44b7d6f6-32d8-4ef4-95bc-eef604fa6b5b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 22:04:14.503704 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:14.493309    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-17d809/premium-w4-17d809-kserve-7c9bdbbcbc-5mtmm" podUID="1e0eca81-7db1-482b-8dd6-5b28fb89e010" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Oct 30 22:04:14.503704 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:14.493478    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5be0d1/premium-w2-5be0d1-kserve-759b6fb995-c6gp4" podUID="933f2470-68de-4768-a05b-6c454f16f5f2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded"
Oct 30 22:04:14.503704 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:14.493731    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-17d809/simulator-w4-17d809-kserve-78d67b6f4-7hsvb" podUID="e7337cf8-4f59-4afb-9b50-6b6c9fa4a773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:14.503704 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:14.493971    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-14af5c/distinct-2-w6-14af5c-kserve-58d554bdb6-r5r6j" podUID="990f4e94-708e-45f3-b2ad-01cd50b3257b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded"
Oct 30 22:04:14.503704 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:14.494221    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5be0d1/distinct-w2-5be0d1-kserve-6b85698c8-sxn9k" podUID="7a2dbfc5-908f-41d1-baff-074c56d2cf16" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 30 22:04:14.503704 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:14.494443    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-17d809/distinct-w4-17d809-kserve-6b8f5c48cd-lhrwp" podUID="ef5e8691-42e1-471b-ba7d-90881028eca7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded"
Oct 30 22:04:14.503704 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:14.494750    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-blrrb"
Oct 30 22:04:14.830702 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:14.830647    5061 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-17d809/distinct-w4-17d809-kserve-6b8f5c48cd-lhrwp" podUID="ef5e8691-42e1-471b-ba7d-90881028eca7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:14.939994 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:14.939948    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-14af5c/distinct-w6-14af5c-kserve-7c7cb9d69c-hljl9" podUID="1f8d9486-fb2f-491a-a8d7-c23c21bd116a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded"
Oct 30 22:04:14.993760 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:14.993714    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/distinct-2-w3-f776cb-kserve-7b4d4cc954-d88pf" podUID="5b184062-7fac-43d4-ba5f-21567df85da3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded"
Oct 30 22:04:15.115958 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:15.115850    5061 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-2d1f91/premium-w0-2d1f91-kserve-6965b558cf-59lgw" podUID="fb84233d-25e0-48bc-8899-c3a3cb3cfeb9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:15.163710 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:15.163651    5061 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-f776cb/distinct-2-w3-f776cb-kserve-7b4d4cc954-d88pf" podUID="5b184062-7fac-43d4-ba5f-21567df85da3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/health\": context deadline exceeded"
Oct 30 22:04:15.295028 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:15.254727    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-14af5c/simulator-w6-14af5c-kserve-85b8bd96d7-wlcb9" podUID="39766c7d-5f1d-42db-8920-c0321cd12c99" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Oct 30 22:04:15.339596 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:15.257664    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2d1f91/premium-w0-2d1f91-kserve-6965b558cf-59lgw" podUID="fb84233d-25e0-48bc-8899-c3a3cb3cfeb9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:15.582552 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:15.520601    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a07620/simulator-w1-a07620-kserve-55b44674f4-4z9lk" podUID="efe7c063-fb0e-4679-8a7c-13f2a0abaa74" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Oct 30 22:04:15.619553 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:15.537595    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-17d809/distinct-w4-17d809-kserve-6b8f5c48cd-lhrwp" podUID="ef5e8691-42e1-471b-ba7d-90881028eca7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:15.916606 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:15.537639    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-blrrb" podUID="cb4b1ed5-2192-4877-8be7-ca82af831af5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": context deadline exceeded"
Oct 30 22:04:15.916606 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:15.537669    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-14af5c/premium-w6-14af5c-kserve-6549647f4-f4gwg" podUID="9b42ee01-c60e-462c-a0e0-b04a52014db6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded"
Oct 30 22:04:15.916606 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:15.537681    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-14af5c/distinct-2-w6-14af5c-kserve-58d554bdb6-r5r6j" podUID="990f4e94-708e-45f3-b2ad-01cd50b3257b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded"
Oct 30 22:04:15.916606 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:15.537693    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5be0d1/premium-w2-5be0d1-kserve-759b6fb995-c6gp4" podUID="933f2470-68de-4768-a05b-6c454f16f5f2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded"
Oct 30 22:04:15.916606 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:15.537704    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-17d809/simulator-w4-17d809-kserve-78d67b6f4-7hsvb" podUID="e7337cf8-4f59-4afb-9b50-6b6c9fa4a773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Oct 30 22:04:15.916606 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:15.554683    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-17d809/premium-w4-17d809-kserve-7c9bdbbcbc-5mtmm" podUID="1e0eca81-7db1-482b-8dd6-5b28fb89e010" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:15.916606 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:15.620677    5061 patch_prober.go:29] interesting pod/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-shwf8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:04:15.916606 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:15.913302    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-shwf8" podUID="9f137aa3-0e26-4526-92b0-2636c979075f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:15.978349 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:15.978283    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5be0d1/distinct-w2-5be0d1-kserve-6b85698c8-sxn9k" podUID="7a2dbfc5-908f-41d1-baff-074c56d2cf16" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 30 22:04:15.983924 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:15.983885    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2d1f91/simulator-w0-2d1f91-kserve-7699765c74-jwtkb" podUID="44b7d6f6-32d8-4ef4-95bc-eef604fa6b5b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 22:04:15.985636 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:15.985600    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-14af5c/distinct-w6-14af5c-kserve-7c7cb9d69c-hljl9" podUID="1f8d9486-fb2f-491a-a8d7-c23c21bd116a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:16.018986 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:16.018656    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/distinct-2-w3-f776cb-kserve-7b4d4cc954-d88pf" podUID="5b184062-7fac-43d4-ba5f-21567df85da3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:16.018986 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:16.018980    5061 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-5be0d1/premium-w2-5be0d1-kserve-759b6fb995-c6gp4" podUID="933f2470-68de-4768-a05b-6c454f16f5f2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": context deadline exceeded"
Oct 30 22:04:16.295609 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:04:16.289373    5061 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.752s"
Oct 30 22:04:16.295609 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:16.292600    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-hfpzd" podUID="8f552c94-29fe-485b-9410-5419467dd6ac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:16.347554 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:16.303165    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-14af5c/simulator-w6-14af5c-kserve-85b8bd96d7-wlcb9" podUID="39766c7d-5f1d-42db-8920-c0321cd12c99" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:16.347554 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:16.318641    5061 patch_prober.go:29] interesting pod/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-txl6d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:04:16.347554 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:16.318709    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-txl6d" podUID="cbaf4a9d-78f3-449b-a2ee-3b7fe4421745" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:16.347554 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:16.318797    5061 patch_prober.go:29] interesting pod/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-zskrq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:04:16.347554 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:16.318820    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-zskrq" podUID="5eeacb87-9f34-4314-80f6-d538c5e843fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:16.580603 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:16.576658    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2d1f91/premium-w0-2d1f91-kserve-6965b558cf-59lgw" podUID="fb84233d-25e0-48bc-8899-c3a3cb3cfeb9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:16.783555 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:16.783012    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42e4d1eb-58l8b"]
Oct 30 22:04:16.837607 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:16.627268    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a07620/simulator-w1-a07620-kserve-55b44674f4-4z9lk" podUID="efe7c063-fb0e-4679-8a7c-13f2a0abaa74" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Oct 30 22:04:16.837607 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:16.819455    5061 cpu_manager.go:414] "containerMap: removing container" podUID="c9665869-1cbf-484c-89e1-d25f21b7ec54" containerName="revoke-keys"
Oct 30 22:04:16.837607 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:16.819486    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9665869-1cbf-484c-89e1-d25f21b7ec54" containerName="revoke-keys"
Oct 30 22:04:16.837607 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:16.819502    5061 cpu_manager.go:414] "containerMap: removing container" podUID="7780bf6d-2d91-4d60-b073-6cc0b1b7009e" containerName="revoke-keys"
Oct 30 22:04:16.837607 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:16.819510    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7780bf6d-2d91-4d60-b073-6cc0b1b7009e" containerName="revoke-keys"
Oct 30 22:04:16.837607 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:16.819861    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9665869-1cbf-484c-89e1-d25f21b7ec54" containerName="revoke-keys"
Oct 30 22:04:16.837607 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:16.819884    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="7780bf6d-2d91-4d60-b073-6cc0b1b7009e" containerName="revoke-keys"
Oct 30 22:04:16.875552 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:16.872013    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42e4d1eb-58l8b"
Oct 30 22:04:16.945698 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:16.945645    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-17d809/premium-w4-17d809-kserve-7c9bdbbcbc-5mtmm" podUID="1e0eca81-7db1-482b-8dd6-5b28fb89e010" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:16.948943 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:16.948908    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-17d809/distinct-w4-17d809-kserve-6b8f5c48cd-lhrwp" podUID="ef5e8691-42e1-471b-ba7d-90881028eca7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:17.001552 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:16.986992    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-17d809/simulator-w4-17d809-kserve-78d67b6f4-7hsvb" podUID="e7337cf8-4f59-4afb-9b50-6b6c9fa4a773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:17.005038 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:17.004997    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-14af5c/distinct-2-w6-14af5c-kserve-58d554bdb6-r5r6j" podUID="990f4e94-708e-45f3-b2ad-01cd50b3257b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded"
Oct 30 22:04:17.006059 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:17.006027    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5be0d1/premium-w2-5be0d1-kserve-759b6fb995-c6gp4" podUID="933f2470-68de-4768-a05b-6c454f16f5f2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded"
Oct 30 22:04:17.018857 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:17.007994    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-blrrb" podUID="cb4b1ed5-2192-4877-8be7-ca82af831af5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:17.018857 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:17.008303    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5be0d1/distinct-w2-5be0d1-kserve-6b85698c8-sxn9k" podUID="7a2dbfc5-908f-41d1-baff-074c56d2cf16" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 30 22:04:17.018857 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:17.007938    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2d1f91/simulator-w0-2d1f91-kserve-7699765c74-jwtkb" podUID="44b7d6f6-32d8-4ef4-95bc-eef604fa6b5b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:17.018857 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:17.017822    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-14af5c/premium-w6-14af5c-kserve-6549647f4-f4gwg" podUID="9b42ee01-c60e-462c-a0e0-b04a52014db6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:17.020907 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:17.020878    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-14af5c/distinct-w6-14af5c-kserve-7c7cb9d69c-hljl9" podUID="1f8d9486-fb2f-491a-a8d7-c23c21bd116a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:17.139605 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:17.122783    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/distinct-2-w3-f776cb-kserve-7b4d4cc954-d88pf" podUID="5b184062-7fac-43d4-ba5f-21567df85da3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded"
Oct 30 22:04:17.264552 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:17.241748    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gdmfg\""
Oct 30 22:04:17.264552 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:17.242388    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:04:17.264552 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:17.250198    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cb4rk\" (UniqueName: \"kubernetes.io/projected/d5fa1ec3-cd5d-4837-a869-e94c66c445cd-kube-api-access-cb4rk\") pod \"maas-api-revoke-sub-42e4d1eb-58l8b\" (UID: \"d5fa1ec3-cd5d-4837-a869-e94c66c445cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42e4d1eb-58l8b"
Oct 30 22:04:17.264552 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:17.250595    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5fa1ec3-cd5d-4837-a869-e94c66c445cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42e4d1eb-58l8b\" (UID: \"d5fa1ec3-cd5d-4837-a869-e94c66c445cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42e4d1eb-58l8b"
Oct 30 22:04:17.264552 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:17.250638    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5fa1ec3-cd5d-4837-a869-e94c66c445cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42e4d1eb-58l8b\" (UID: \"d5fa1ec3-cd5d-4837-a869-e94c66c445cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42e4d1eb-58l8b"
Oct 30 22:04:17.303823 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:17.303783    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-14af5c/simulator-w6-14af5c-kserve-85b8bd96d7-wlcb9" podUID="39766c7d-5f1d-42db-8920-c0321cd12c99" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:17.314740 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:17.314667    5061 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-blrrb" podUID="cb4b1ed5-2192-4877-8be7-ca82af831af5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:17.451103 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:17.451001    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cb4rk\" (UniqueName: \"kubernetes.io/projected/d5fa1ec3-cd5d-4837-a869-e94c66c445cd-kube-api-access-cb4rk\") pod \"maas-api-revoke-sub-42e4d1eb-58l8b\" (UID: \"d5fa1ec3-cd5d-4837-a869-e94c66c445cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42e4d1eb-58l8b"
Oct 30 22:04:17.472974 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:17.472931    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5fa1ec3-cd5d-4837-a869-e94c66c445cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42e4d1eb-58l8b\" (UID: \"d5fa1ec3-cd5d-4837-a869-e94c66c445cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42e4d1eb-58l8b"
Oct 30 22:04:17.473249 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:17.473226    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5fa1ec3-cd5d-4837-a869-e94c66c445cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42e4d1eb-58l8b\" (UID: \"d5fa1ec3-cd5d-4837-a869-e94c66c445cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42e4d1eb-58l8b"
Oct 30 22:04:17.539985 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:17.539946    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5fa1ec3-cd5d-4837-a869-e94c66c445cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42e4d1eb-58l8b\" (UID: \"d5fa1ec3-cd5d-4837-a869-e94c66c445cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42e4d1eb-58l8b"
Oct 30 22:04:17.633826 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:17.633790    5061 patch_prober.go:29] interesting pod/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5zvwr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:04:17.634088 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:17.634056    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5zvwr" podUID="3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:17.635476 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:17.635447    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5fa1ec3-cd5d-4837-a869-e94c66c445cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42e4d1eb-58l8b\" (UID: \"d5fa1ec3-cd5d-4837-a869-e94c66c445cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42e4d1eb-58l8b"
Oct 30 22:04:17.651964 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:17.651913    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2d1f91/premium-w0-2d1f91-kserve-6965b558cf-59lgw" podUID="fb84233d-25e0-48bc-8899-c3a3cb3cfeb9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:17.804558 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:17.799700    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42e4d1eb-58l8b"]
Oct 30 22:04:17.820377 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:17.820337    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a07620/simulator-w1-a07620-kserve-55b44674f4-4z9lk" podUID="efe7c063-fb0e-4679-8a7c-13f2a0abaa74" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:17.864550 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:17.824817    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cb4rk\" (UniqueName: \"kubernetes.io/projected/d5fa1ec3-cd5d-4837-a869-e94c66c445cd-kube-api-access-cb4rk\") pod \"maas-api-revoke-sub-42e4d1eb-58l8b\" (UID: \"d5fa1ec3-cd5d-4837-a869-e94c66c445cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42e4d1eb-58l8b"
Oct 30 22:04:17.901132 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:17.901073    5061 patch_prober.go:29] interesting pod/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-4rghz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:04:17.901462 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:17.901434    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-4rghz" podUID="b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:17.950406 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:17.950378    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-17d809/simulator-w4-17d809-kserve-78d67b6f4-7hsvb"
Oct 30 22:04:17.959108 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:17.959063    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-17d809/distinct-w4-17d809-kserve-6b8f5c48cd-lhrwp" podUID="ef5e8691-42e1-471b-ba7d-90881028eca7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded"
Oct 30 22:04:17.963950 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:17.963912    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-17d809/premium-w4-17d809-kserve-7c9bdbbcbc-5mtmm" podUID="1e0eca81-7db1-482b-8dd6-5b28fb89e010" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Oct 30 22:04:17.987762 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:17.987732    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-14af5c/premium-w6-14af5c-kserve-6549647f4-f4gwg"
Oct 30 22:04:18.034668 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:18.034618    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5be0d1/distinct-w2-5be0d1-kserve-6b85698c8-sxn9k" podUID="7a2dbfc5-908f-41d1-baff-074c56d2cf16" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:18.035260 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:18.035223    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-blrrb" podUID="cb4b1ed5-2192-4877-8be7-ca82af831af5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": context deadline exceeded"
Oct 30 22:04:18.046547 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:18.038384    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-14af5c/distinct-w6-14af5c-kserve-7c7cb9d69c-hljl9" podUID="1f8d9486-fb2f-491a-a8d7-c23c21bd116a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:18.046547 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:18.039158    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-14af5c/distinct-2-w6-14af5c-kserve-58d554bdb6-r5r6j" podUID="990f4e94-708e-45f3-b2ad-01cd50b3257b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:18.046547 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:18.039778    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5be0d1/premium-w2-5be0d1-kserve-759b6fb995-c6gp4" podUID="933f2470-68de-4768-a05b-6c454f16f5f2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:18.116641 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:18.111443    5061 patch_prober.go:29] interesting pod/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-f95pk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:04:18.127231 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:18.127181    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-f95pk" podUID="081e86b3-a830-4a91-a7da-c270baca6eee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:18.146131 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:18.146091    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/distinct-2-w3-f776cb-kserve-7b4d4cc954-d88pf" podUID="5b184062-7fac-43d4-ba5f-21567df85da3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:18.258588 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:18.256816    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2d1f91/simulator-w0-2d1f91-kserve-7699765c74-jwtkb" podUID="44b7d6f6-32d8-4ef4-95bc-eef604fa6b5b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:18.265449 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:18.263902    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42e4d1eb-58l8b"
Oct 30 22:04:18.275371 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:18.275349    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2d1f91/premium-w0-2d1f91-kserve-6965b558cf-59lgw"
Oct 30 22:04:18.276870 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:18.276848    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-14af5c/simulator-w6-14af5c-kserve-85b8bd96d7-wlcb9"
Oct 30 22:04:18.280371 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:18.278498    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-17d809/premium-w4-17d809-kserve-7c9bdbbcbc-5mtmm"
Oct 30 22:04:18.289847 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:18.289824    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f776cb/distinct-2-w3-f776cb-kserve-7b4d4cc954-d88pf"
Oct 30 22:04:18.300712 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:18.300688    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-14af5c/distinct-w6-14af5c-kserve-7c7cb9d69c-hljl9"
Oct 30 22:04:18.300915 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:18.300904    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a07620/simulator-w1-a07620-kserve-55b44674f4-4z9lk"
Oct 30 22:04:18.334643 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:18.334611    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-17d809/distinct-w4-17d809-kserve-6b8f5c48cd-lhrwp"
Oct 30 22:04:18.334959 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:18.334940    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-blrrb"
Oct 30 22:04:18.395437 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:18.395361    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-5be0d1/distinct-w2-5be0d1-kserve-6b85698c8-sxn9k"
Oct 30 22:04:18.399130 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:18.399101    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-14af5c/distinct-2-w6-14af5c-kserve-58d554bdb6-r5r6j"
Oct 30 22:04:18.403203 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:18.403138    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2d1f91/simulator-w0-2d1f91-kserve-7699765c74-jwtkb"
Oct 30 22:04:18.403624 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:18.403560    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-5be0d1/premium-w2-5be0d1-kserve-759b6fb995-c6gp4"
Oct 30 22:04:18.748965 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:18.748937    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42e4d1eb-58l8b"]
Oct 30 22:04:18.750732 ip-10-0-130-60 kubenswrapper[5061]: W1030 22:04:18.750703    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5fa1ec3_cd5d_4837_a869_e94c66c445cd.slice/crio-0f2b9a58579050d676d71b40c6f736db76ac40d7c456774e01722367c1fef354 WatchSource:0}: Error finding container 0f2b9a58579050d676d71b40c6f736db76ac40d7c456774e01722367c1fef354: Status 404 returned error can't find the container with id 0f2b9a58579050d676d71b40c6f736db76ac40d7c456774e01722367c1fef354
Oct 30 22:04:19.030962 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:19.030815    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42e4d1eb-58l8b" event={"ID":"d5fa1ec3-cd5d-4837-a869-e94c66c445cd","Type":"ContainerStarted","Data":"0f2b9a58579050d676d71b40c6f736db76ac40d7c456774e01722367c1fef354"}
Oct 30 22:04:20.041279 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:20.041184    5061 generic.go:356] "Generic (PLEG): container finished" podID="d5fa1ec3-cd5d-4837-a869-e94c66c445cd" containerID="e96c9b2c27304c01e32242a5692a786ba88659d68eb97979ccee5f44c627e9a2" exitCode=0
Oct 30 22:04:20.041279 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:20.041217    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42e4d1eb-58l8b" event={"ID":"d5fa1ec3-cd5d-4837-a869-e94c66c445cd","Type":"ContainerDied","Data":"e96c9b2c27304c01e32242a5692a786ba88659d68eb97979ccee5f44c627e9a2"}
Oct 30 22:04:20.945213 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:20.945182    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-998abd03-txgxm"]
Oct 30 22:04:20.949768 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:20.949747    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-998abd03-txgxm"
Oct 30 22:04:21.005606 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:21.005579    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-998abd03-txgxm"]
Oct 30 22:04:21.037971 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:21.037932    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cad0f17c-1e87-406f-848d-d01ea9ba046f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-998abd03-txgxm\" (UID: \"cad0f17c-1e87-406f-848d-d01ea9ba046f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-998abd03-txgxm"
Oct 30 22:04:21.038158 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:21.038065    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vlk8\" (UniqueName: \"kubernetes.io/projected/cad0f17c-1e87-406f-848d-d01ea9ba046f-kube-api-access-2vlk8\") pod \"maas-api-revoke-sub-998abd03-txgxm\" (UID: \"cad0f17c-1e87-406f-848d-d01ea9ba046f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-998abd03-txgxm"
Oct 30 22:04:21.038342 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:21.038304    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cad0f17c-1e87-406f-848d-d01ea9ba046f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-998abd03-txgxm\" (UID: \"cad0f17c-1e87-406f-848d-d01ea9ba046f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-998abd03-txgxm"
Oct 30 22:04:21.140356 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:21.140316    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cad0f17c-1e87-406f-848d-d01ea9ba046f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-998abd03-txgxm\" (UID: \"cad0f17c-1e87-406f-848d-d01ea9ba046f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-998abd03-txgxm"
Oct 30 22:04:21.141074 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:21.141050    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cad0f17c-1e87-406f-848d-d01ea9ba046f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-998abd03-txgxm\" (UID: \"cad0f17c-1e87-406f-848d-d01ea9ba046f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-998abd03-txgxm"
Oct 30 22:04:21.141296 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:21.141282    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vlk8\" (UniqueName: \"kubernetes.io/projected/cad0f17c-1e87-406f-848d-d01ea9ba046f-kube-api-access-2vlk8\") pod \"maas-api-revoke-sub-998abd03-txgxm\" (UID: \"cad0f17c-1e87-406f-848d-d01ea9ba046f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-998abd03-txgxm"
Oct 30 22:04:21.141947 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:21.141889    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cad0f17c-1e87-406f-848d-d01ea9ba046f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-998abd03-txgxm\" (UID: \"cad0f17c-1e87-406f-848d-d01ea9ba046f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-998abd03-txgxm"
Oct 30 22:04:21.147447 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:21.147417    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cad0f17c-1e87-406f-848d-d01ea9ba046f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-998abd03-txgxm\" (UID: \"cad0f17c-1e87-406f-848d-d01ea9ba046f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-998abd03-txgxm"
Oct 30 22:04:21.197122 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:21.197035    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vlk8\" (UniqueName: \"kubernetes.io/projected/cad0f17c-1e87-406f-848d-d01ea9ba046f-kube-api-access-2vlk8\") pod \"maas-api-revoke-sub-998abd03-txgxm\" (UID: \"cad0f17c-1e87-406f-848d-d01ea9ba046f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-998abd03-txgxm"
Oct 30 22:04:21.253627 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:21.253601    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42e4d1eb-58l8b"
Oct 30 22:04:21.263912 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:21.263892    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-998abd03-txgxm"
Oct 30 22:04:21.344195 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:21.344169    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5fa1ec3-cd5d-4837-a869-e94c66c445cd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5fa1ec3-cd5d-4837-a869-e94c66c445cd-maas-api-openshift-ca\") pod \"d5fa1ec3-cd5d-4837-a869-e94c66c445cd\" (UID: \"d5fa1ec3-cd5d-4837-a869-e94c66c445cd\") "
Oct 30 22:04:21.344322 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:21.344230    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5fa1ec3-cd5d-4837-a869-e94c66c445cd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5fa1ec3-cd5d-4837-a869-e94c66c445cd-maas-api-xks-ca\") pod \"d5fa1ec3-cd5d-4837-a869-e94c66c445cd\" (UID: \"d5fa1ec3-cd5d-4837-a869-e94c66c445cd\") "
Oct 30 22:04:21.344322 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:21.344276    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5fa1ec3-cd5d-4837-a869-e94c66c445cd-kube-api-access-cb4rk\" (UniqueName: \"kubernetes.io/projected/d5fa1ec3-cd5d-4837-a869-e94c66c445cd-kube-api-access-cb4rk\") pod \"d5fa1ec3-cd5d-4837-a869-e94c66c445cd\" (UID: \"d5fa1ec3-cd5d-4837-a869-e94c66c445cd\") "
Oct 30 22:04:21.344626 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:21.344595    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5fa1ec3-cd5d-4837-a869-e94c66c445cd-maas-api-openshift-ca" pod "d5fa1ec3-cd5d-4837-a869-e94c66c445cd" (UID: "d5fa1ec3-cd5d-4837-a869-e94c66c445cd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:04:21.344999 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:21.344977    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5fa1ec3-cd5d-4837-a869-e94c66c445cd-maas-api-openshift-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:04:21.347425 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:21.347398    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5fa1ec3-cd5d-4837-a869-e94c66c445cd-maas-api-xks-ca" pod "d5fa1ec3-cd5d-4837-a869-e94c66c445cd" (UID: "d5fa1ec3-cd5d-4837-a869-e94c66c445cd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:04:21.347559 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:21.347426    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5fa1ec3-cd5d-4837-a869-e94c66c445cd-kube-api-access-cb4rk" pod "d5fa1ec3-cd5d-4837-a869-e94c66c445cd" (UID: "d5fa1ec3-cd5d-4837-a869-e94c66c445cd"). InnerVolumeSpecName "kube-api-access-cb4rk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:04:21.446389 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:21.446358    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5fa1ec3-cd5d-4837-a869-e94c66c445cd-maas-api-xks-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:04:21.446389 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:21.446386    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cb4rk\" (UniqueName: \"kubernetes.io/projected/d5fa1ec3-cd5d-4837-a869-e94c66c445cd-kube-api-access-cb4rk\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:04:21.637706 ip-10-0-130-60 kubenswrapper[5061]: W1030 22:04:21.637672    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcad0f17c_1e87_406f_848d_d01ea9ba046f.slice/crio-29b60ef63ad2ce1c1df7ee5c2107b218ba308f8c246d86d922944444b2a60646 WatchSource:0}: Error finding container 29b60ef63ad2ce1c1df7ee5c2107b218ba308f8c246d86d922944444b2a60646: Status 404 returned error can't find the container with id 29b60ef63ad2ce1c1df7ee5c2107b218ba308f8c246d86d922944444b2a60646
Oct 30 22:04:21.638148 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:21.638125    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-998abd03-txgxm"]
Oct 30 22:04:22.059457 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:22.059428    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42e4d1eb-58l8b"
Oct 30 22:04:22.059457 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:22.059446    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42e4d1eb-58l8b" event={"ID":"d5fa1ec3-cd5d-4837-a869-e94c66c445cd","Type":"ContainerDied","Data":"0f2b9a58579050d676d71b40c6f736db76ac40d7c456774e01722367c1fef354"}
Oct 30 22:04:22.059733 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:22.059514    5061 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f2b9a58579050d676d71b40c6f736db76ac40d7c456774e01722367c1fef354"
Oct 30 22:04:22.061175 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:22.061141    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-998abd03-txgxm" event={"ID":"cad0f17c-1e87-406f-848d-d01ea9ba046f","Type":"ContainerStarted","Data":"79942dbff1f6f58c04e91105bf0ca84032a3f559815eef93a7e9d01221aa97d0"}
Oct 30 22:04:22.061294 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:22.061191    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-998abd03-txgxm" event={"ID":"cad0f17c-1e87-406f-848d-d01ea9ba046f","Type":"ContainerStarted","Data":"29b60ef63ad2ce1c1df7ee5c2107b218ba308f8c246d86d922944444b2a60646"}
Oct 30 22:04:22.171675 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:22.171578    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-998abd03-txgxm" podStartSLOduration=2.17156035 podStartE2EDuration="2.17156035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:22.169106135 +0000 UTC m=+1315.781875924" watchObservedRunningTime="2026-10-30 22:04:22.17156035 +0000 UTC m=+1315.784330108"
Oct 30 22:04:23.071420 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:23.071377    5061 generic.go:356] "Generic (PLEG): container finished" podID="cad0f17c-1e87-406f-848d-d01ea9ba046f" containerID="79942dbff1f6f58c04e91105bf0ca84032a3f559815eef93a7e9d01221aa97d0" exitCode=0
Oct 30 22:04:23.071675 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:23.071468    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-998abd03-txgxm" event={"ID":"cad0f17c-1e87-406f-848d-d01ea9ba046f","Type":"ContainerDied","Data":"79942dbff1f6f58c04e91105bf0ca84032a3f559815eef93a7e9d01221aa97d0"}
Oct 30 22:04:24.276581 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:24.276543    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-998abd03-txgxm"
Oct 30 22:04:24.389436 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:24.389353    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cad0f17c-1e87-406f-848d-d01ea9ba046f-kube-api-access-2vlk8\" (UniqueName: \"kubernetes.io/projected/cad0f17c-1e87-406f-848d-d01ea9ba046f-kube-api-access-2vlk8\") pod \"cad0f17c-1e87-406f-848d-d01ea9ba046f\" (UID: \"cad0f17c-1e87-406f-848d-d01ea9ba046f\") "
Oct 30 22:04:24.389436 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:24.389395    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cad0f17c-1e87-406f-848d-d01ea9ba046f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cad0f17c-1e87-406f-848d-d01ea9ba046f-maas-api-openshift-ca\") pod \"cad0f17c-1e87-406f-848d-d01ea9ba046f\" (UID: \"cad0f17c-1e87-406f-848d-d01ea9ba046f\") "
Oct 30 22:04:24.389701 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:24.389628    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cad0f17c-1e87-406f-848d-d01ea9ba046f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cad0f17c-1e87-406f-848d-d01ea9ba046f-maas-api-xks-ca\") pod \"cad0f17c-1e87-406f-848d-d01ea9ba046f\" (UID: \"cad0f17c-1e87-406f-848d-d01ea9ba046f\") "
Oct 30 22:04:24.389873 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:24.389846    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cad0f17c-1e87-406f-848d-d01ea9ba046f-maas-api-openshift-ca" pod "cad0f17c-1e87-406f-848d-d01ea9ba046f" (UID: "cad0f17c-1e87-406f-848d-d01ea9ba046f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:04:24.390479 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:24.390451    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cad0f17c-1e87-406f-848d-d01ea9ba046f-maas-api-openshift-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:04:24.392228 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:24.392203    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cad0f17c-1e87-406f-848d-d01ea9ba046f-kube-api-access-2vlk8" pod "cad0f17c-1e87-406f-848d-d01ea9ba046f" (UID: "cad0f17c-1e87-406f-848d-d01ea9ba046f"). InnerVolumeSpecName "kube-api-access-2vlk8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:04:24.392433 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:24.392336    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cad0f17c-1e87-406f-848d-d01ea9ba046f-maas-api-xks-ca" pod "cad0f17c-1e87-406f-848d-d01ea9ba046f" (UID: "cad0f17c-1e87-406f-848d-d01ea9ba046f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:04:24.491855 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:24.491818    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vlk8\" (UniqueName: \"kubernetes.io/projected/cad0f17c-1e87-406f-848d-d01ea9ba046f-kube-api-access-2vlk8\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:04:24.491855 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:24.491850    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cad0f17c-1e87-406f-848d-d01ea9ba046f-maas-api-xks-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:04:24.667786 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:04:24.667705    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 22:04:25.089171 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:25.089133    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-998abd03-txgxm" event={"ID":"cad0f17c-1e87-406f-848d-d01ea9ba046f","Type":"ContainerDied","Data":"29b60ef63ad2ce1c1df7ee5c2107b218ba308f8c246d86d922944444b2a60646"}
Oct 30 22:04:25.089336 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:25.089181    5061 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="29b60ef63ad2ce1c1df7ee5c2107b218ba308f8c246d86d922944444b2a60646"
Oct 30 22:04:25.089336 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:25.089147    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-998abd03-txgxm"
Oct 30 22:04:26.810095 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:26.810066    5061 scope.go:122] "RemoveContainer" containerID="f40061f3bb6d6d4ae40d5f5c8b7be5223439bfb08a88c1711c2963bce852e263"
Oct 30 22:04:26.820268 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:26.820245    5061 scope.go:122] "RemoveContainer" containerID="811260f68d964cd95694cf250e29fcf0451ff7ee7de171e40a91454a01111ece"
Oct 30 22:04:31.309318 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:31.309286    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51bdef55-pvst4"]
Oct 30 22:04:31.309957 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:31.309942    5061 cpu_manager.go:414] "containerMap: removing container" podUID="cad0f17c-1e87-406f-848d-d01ea9ba046f" containerName="revoke-keys"
Oct 30 22:04:31.310003 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:31.309958    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cad0f17c-1e87-406f-848d-d01ea9ba046f" containerName="revoke-keys"
Oct 30 22:04:31.310003 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:31.309972    5061 cpu_manager.go:414] "containerMap: removing container" podUID="d5fa1ec3-cd5d-4837-a869-e94c66c445cd" containerName="revoke-keys"
Oct 30 22:04:31.310003 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:31.309977    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5fa1ec3-cd5d-4837-a869-e94c66c445cd" containerName="revoke-keys"
Oct 30 22:04:31.310115 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:31.310105    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="cad0f17c-1e87-406f-848d-d01ea9ba046f" containerName="revoke-keys"
Oct 30 22:04:31.310155 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:31.310118    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5fa1ec3-cd5d-4837-a869-e94c66c445cd" containerName="revoke-keys"
Oct 30 22:04:31.315213 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:31.315185    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51bdef55-pvst4"
Oct 30 22:04:31.369088 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:31.369059    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51bdef55-pvst4"]
Oct 30 22:04:31.393664 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:31.393627    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7c1a2ce-12a9-4e6a-8586-5d39f62059fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51bdef55-pvst4\" (UID: \"c7c1a2ce-12a9-4e6a-8586-5d39f62059fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51bdef55-pvst4"
Oct 30 22:04:31.393810 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:31.393681    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4glhh\" (UniqueName: \"kubernetes.io/projected/c7c1a2ce-12a9-4e6a-8586-5d39f62059fa-kube-api-access-4glhh\") pod \"maas-api-revoke-sub-51bdef55-pvst4\" (UID: \"c7c1a2ce-12a9-4e6a-8586-5d39f62059fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51bdef55-pvst4"
Oct 30 22:04:31.393939 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:31.393910    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7c1a2ce-12a9-4e6a-8586-5d39f62059fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51bdef55-pvst4\" (UID: \"c7c1a2ce-12a9-4e6a-8586-5d39f62059fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51bdef55-pvst4"
Oct 30 22:04:31.411379 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:31.411352    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gdmfg\""
Oct 30 22:04:31.411379 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:31.411353    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:04:31.495053 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:31.495009    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7c1a2ce-12a9-4e6a-8586-5d39f62059fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51bdef55-pvst4\" (UID: \"c7c1a2ce-12a9-4e6a-8586-5d39f62059fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51bdef55-pvst4"
Oct 30 22:04:31.495224 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:31.495080    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4glhh\" (UniqueName: \"kubernetes.io/projected/c7c1a2ce-12a9-4e6a-8586-5d39f62059fa-kube-api-access-4glhh\") pod \"maas-api-revoke-sub-51bdef55-pvst4\" (UID: \"c7c1a2ce-12a9-4e6a-8586-5d39f62059fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51bdef55-pvst4"
Oct 30 22:04:31.495340 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:31.495269    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7c1a2ce-12a9-4e6a-8586-5d39f62059fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51bdef55-pvst4\" (UID: \"c7c1a2ce-12a9-4e6a-8586-5d39f62059fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51bdef55-pvst4"
Oct 30 22:04:31.496163 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:31.496137    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7c1a2ce-12a9-4e6a-8586-5d39f62059fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51bdef55-pvst4\" (UID: \"c7c1a2ce-12a9-4e6a-8586-5d39f62059fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51bdef55-pvst4"
Oct 30 22:04:31.497836 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:31.497809    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7c1a2ce-12a9-4e6a-8586-5d39f62059fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51bdef55-pvst4\" (UID: \"c7c1a2ce-12a9-4e6a-8586-5d39f62059fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51bdef55-pvst4"
Oct 30 22:04:31.549422 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:31.549393    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4glhh\" (UniqueName: \"kubernetes.io/projected/c7c1a2ce-12a9-4e6a-8586-5d39f62059fa-kube-api-access-4glhh\") pod \"maas-api-revoke-sub-51bdef55-pvst4\" (UID: \"c7c1a2ce-12a9-4e6a-8586-5d39f62059fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51bdef55-pvst4"
Oct 30 22:04:31.629288 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:31.629216    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51bdef55-pvst4"
Oct 30 22:04:32.000269 ip-10-0-130-60 kubenswrapper[5061]: W1030 22:04:32.000234    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7c1a2ce_12a9_4e6a_8586_5d39f62059fa.slice/crio-65e24279041de148ca34598d07d7409254822ea708c51e24635360444046bd73 WatchSource:0}: Error finding container 65e24279041de148ca34598d07d7409254822ea708c51e24635360444046bd73: Status 404 returned error can't find the container with id 65e24279041de148ca34598d07d7409254822ea708c51e24635360444046bd73
Oct 30 22:04:32.000413 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:32.000345    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51bdef55-pvst4"]
Oct 30 22:04:32.143085 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:32.143052    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51bdef55-pvst4" event={"ID":"c7c1a2ce-12a9-4e6a-8586-5d39f62059fa","Type":"ContainerStarted","Data":"65e24279041de148ca34598d07d7409254822ea708c51e24635360444046bd73"}
Oct 30 22:04:33.153093 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:33.153062    5061 generic.go:356] "Generic (PLEG): container finished" podID="c7c1a2ce-12a9-4e6a-8586-5d39f62059fa" containerID="843e34dea9f18cdfa8d420a37f29e87ace6d8c307d1bca6247e223d1f3571283" exitCode=0
Oct 30 22:04:33.153458 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:33.153149    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51bdef55-pvst4" event={"ID":"c7c1a2ce-12a9-4e6a-8586-5d39f62059fa","Type":"ContainerDied","Data":"843e34dea9f18cdfa8d420a37f29e87ace6d8c307d1bca6247e223d1f3571283"}
Oct 30 22:04:34.309828 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:34.309792    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7185-215e51"]
Oct 30 22:04:34.315315 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:34.315294    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7185-215e51"
Oct 30 22:04:34.342977 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:34.342954    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51bdef55-pvst4"
Oct 30 22:04:34.367191 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:34.367164    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7185-215e51"]
Oct 30 22:04:34.410186 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:34.410156    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-74ld5\""
Oct 30 22:04:34.410347 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:34.410186    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Oct 30 22:04:34.411853 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:34.411796    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Oct 30 22:04:34.452392 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:34.452370    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7c1a2ce-12a9-4e6a-8586-5d39f62059fa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7c1a2ce-12a9-4e6a-8586-5d39f62059fa-maas-api-openshift-ca\") pod \"c7c1a2ce-12a9-4e6a-8586-5d39f62059fa\" (UID: \"c7c1a2ce-12a9-4e6a-8586-5d39f62059fa\") "
Oct 30 22:04:34.452516 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:34.452499    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7c1a2ce-12a9-4e6a-8586-5d39f62059fa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7c1a2ce-12a9-4e6a-8586-5d39f62059fa-maas-api-xks-ca\") pod \"c7c1a2ce-12a9-4e6a-8586-5d39f62059fa\" (UID: \"c7c1a2ce-12a9-4e6a-8586-5d39f62059fa\") "
Oct 30 22:04:34.452714 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:34.452696    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7c1a2ce-12a9-4e6a-8586-5d39f62059fa-kube-api-access-4glhh\" (UniqueName: \"kubernetes.io/projected/c7c1a2ce-12a9-4e6a-8586-5d39f62059fa-kube-api-access-4glhh\") pod \"c7c1a2ce-12a9-4e6a-8586-5d39f62059fa\" (UID: \"c7c1a2ce-12a9-4e6a-8586-5d39f62059fa\") "
Oct 30 22:04:34.452750 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:34.452724    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7c1a2ce-12a9-4e6a-8586-5d39f62059fa-maas-api-openshift-ca" pod "c7c1a2ce-12a9-4e6a-8586-5d39f62059fa" (UID: "c7c1a2ce-12a9-4e6a-8586-5d39f62059fa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:04:34.453158 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:34.453134    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfq8d\" (UniqueName: \"kubernetes.io/projected/cdd4b251-9868-4b9c-ad82-8c913536950b-kube-api-access-lfq8d\") pod \"test-curl-7185-215e51\" (UID: \"cdd4b251-9868-4b9c-ad82-8c913536950b\") " pod="opendatahub/test-curl-7185-215e51"
Oct 30 22:04:34.453387 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:34.453363    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7c1a2ce-12a9-4e6a-8586-5d39f62059fa-maas-api-openshift-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:04:34.454728 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:34.454707    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7c1a2ce-12a9-4e6a-8586-5d39f62059fa-kube-api-access-4glhh" pod "c7c1a2ce-12a9-4e6a-8586-5d39f62059fa" (UID: "c7c1a2ce-12a9-4e6a-8586-5d39f62059fa"). InnerVolumeSpecName "kube-api-access-4glhh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:04:34.454801 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:34.454750    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7c1a2ce-12a9-4e6a-8586-5d39f62059fa-maas-api-xks-ca" pod "c7c1a2ce-12a9-4e6a-8586-5d39f62059fa" (UID: "c7c1a2ce-12a9-4e6a-8586-5d39f62059fa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:04:34.554361 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:34.554322    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfq8d\" (UniqueName: \"kubernetes.io/projected/cdd4b251-9868-4b9c-ad82-8c913536950b-kube-api-access-lfq8d\") pod \"test-curl-7185-215e51\" (UID: \"cdd4b251-9868-4b9c-ad82-8c913536950b\") " pod="opendatahub/test-curl-7185-215e51"
Oct 30 22:04:34.554506 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:34.554423    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7c1a2ce-12a9-4e6a-8586-5d39f62059fa-maas-api-xks-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:04:34.554506 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:34.554437    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4glhh\" (UniqueName: \"kubernetes.io/projected/c7c1a2ce-12a9-4e6a-8586-5d39f62059fa-kube-api-access-4glhh\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:04:34.608498 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:34.608469    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfq8d\" (UniqueName: \"kubernetes.io/projected/cdd4b251-9868-4b9c-ad82-8c913536950b-kube-api-access-lfq8d\") pod \"test-curl-7185-215e51\" (UID: \"cdd4b251-9868-4b9c-ad82-8c913536950b\") " pod="opendatahub/test-curl-7185-215e51"
Oct 30 22:04:34.650886 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:34.650853    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7185-215e51"
Oct 30 22:04:35.027681 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:35.027607    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7185-215e51"]
Oct 30 22:04:35.028355 ip-10-0-130-60 kubenswrapper[5061]: W1030 22:04:35.028326    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcdd4b251_9868_4b9c_ad82_8c913536950b.slice/crio-94bcf8631eeb62d75e32bb2289d3f91185a3b03c345cf0a84da88a4e030e9c6b WatchSource:0}: Error finding container 94bcf8631eeb62d75e32bb2289d3f91185a3b03c345cf0a84da88a4e030e9c6b: Status 404 returned error can't find the container with id 94bcf8631eeb62d75e32bb2289d3f91185a3b03c345cf0a84da88a4e030e9c6b
Oct 30 22:04:35.030351 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:35.030334    5061 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 22:04:35.169002 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:35.168967    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7185-215e51" event={"ID":"cdd4b251-9868-4b9c-ad82-8c913536950b","Type":"ContainerStarted","Data":"94bcf8631eeb62d75e32bb2289d3f91185a3b03c345cf0a84da88a4e030e9c6b"}
Oct 30 22:04:35.170237 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:35.170217    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51bdef55-pvst4"
Oct 30 22:04:35.170354 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:35.170246    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51bdef55-pvst4" event={"ID":"c7c1a2ce-12a9-4e6a-8586-5d39f62059fa","Type":"ContainerDied","Data":"65e24279041de148ca34598d07d7409254822ea708c51e24635360444046bd73"}
Oct 30 22:04:35.170354 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:35.170296    5061 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="65e24279041de148ca34598d07d7409254822ea708c51e24635360444046bd73"
Oct 30 22:04:38.199147 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:38.199108    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7185-215e51" event={"ID":"cdd4b251-9868-4b9c-ad82-8c913536950b","Type":"ContainerStarted","Data":"3e196b6b24093386aef9c49ab1fec521432446176581c5dace607c76821b4e2c"}
Oct 30 22:04:38.307666 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:38.307619    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7185-215e51" podStartSLOduration=1.7616237959999999 podStartE2EDuration="4.30760555s" totalImagesPullingTime="2.545981754s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:38.304303107 +0000 UTC m=+1331.917072866" watchObservedRunningTime="2026-10-30 22:04:38.30760555 +0000 UTC m=+1331.920375307"
Oct 30 22:04:38.384225 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:38.384195    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-b3084d7f-openshift-default-5b87dddbbf-5h4fs"]
Oct 30 22:04:38.384888 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:38.384872    5061 cpu_manager.go:414] "containerMap: removing container" podUID="c7c1a2ce-12a9-4e6a-8586-5d39f62059fa" containerName="revoke-keys"
Oct 30 22:04:38.384949 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:38.384889    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7c1a2ce-12a9-4e6a-8586-5d39f62059fa" containerName="revoke-keys"
Oct 30 22:04:38.385027 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:38.385018    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7c1a2ce-12a9-4e6a-8586-5d39f62059fa" containerName="revoke-keys"
Oct 30 22:04:38.388716 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:38.388699    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b3084d7f-openshift-default-5b87dddbbf-5h4fs"
Oct 30 22:04:38.453229 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:38.452777    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-b3084d7f-openshift-default-5b87dddbbf-5h4fs"]
Oct 30 22:04:38.542921 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:38.535152    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14ca670b-dd10-4d17-99d0-b1497d64da96-workload-socket\") pod \"e2e-ait-b3084d7f-openshift-default-5b87dddbbf-5h4fs\" (UID: \"14ca670b-dd10-4d17-99d0-b1497d64da96\") " pod="openshift-ingress/e2e-ait-b3084d7f-openshift-default-5b87dddbbf-5h4fs"
Oct 30 22:04:38.542921 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:38.535481    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14ca670b-dd10-4d17-99d0-b1497d64da96-istio-token\") pod \"e2e-ait-b3084d7f-openshift-default-5b87dddbbf-5h4fs\" (UID: \"14ca670b-dd10-4d17-99d0-b1497d64da96\") " pod="openshift-ingress/e2e-ait-b3084d7f-openshift-default-5b87dddbbf-5h4fs"
Oct 30 22:04:38.542921 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:38.535596    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14ca670b-dd10-4d17-99d0-b1497d64da96-istio-podinfo\") pod \"e2e-ait-b3084d7f-openshift-default-5b87dddbbf-5h4fs\" (UID: \"14ca670b-dd10-4d17-99d0-b1497d64da96\") " pod="openshift-ingress/e2e-ait-b3084d7f-openshift-default-5b87dddbbf-5h4fs"
Oct 30 22:04:38.542921 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:38.535659    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bht8\" (UniqueName: \"kubernetes.io/projected/14ca670b-dd10-4d17-99d0-b1497d64da96-kube-api-access-7bht8\") pod \"e2e-ait-b3084d7f-openshift-default-5b87dddbbf-5h4fs\" (UID: \"14ca670b-dd10-4d17-99d0-b1497d64da96\") " pod="openshift-ingress/e2e-ait-b3084d7f-openshift-default-5b87dddbbf-5h4fs"
Oct 30 22:04:38.542921 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:38.535882    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14ca670b-dd10-4d17-99d0-b1497d64da96-workload-certs\") pod \"e2e-ait-b3084d7f-openshift-default-5b87dddbbf-5h4fs\" (UID: \"14ca670b-dd10-4d17-99d0-b1497d64da96\") " pod="openshift-ingress/e2e-ait-b3084d7f-openshift-default-5b87dddbbf-5h4fs"
Oct 30 22:04:38.542921 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:38.535909    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14ca670b-dd10-4d17-99d0-b1497d64da96-istio-data\") pod \"e2e-ait-b3084d7f-openshift-default-5b87dddbbf-5h4fs\" (UID: \"14ca670b-dd10-4d17-99d0-b1497d64da96\") " pod="openshift-ingress/e2e-ait-b3084d7f-openshift-default-5b87dddbbf-5h4fs"
Oct 30 22:04:38.542921 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:38.535997    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14ca670b-dd10-4d17-99d0-b1497d64da96-credential-socket\") pod \"e2e-ait-b3084d7f-openshift-default-5b87dddbbf-5h4fs\" (UID: \"14ca670b-dd10-4d17-99d0-b1497d64da96\") " pod="openshift-ingress/e2e-ait-b3084d7f-openshift-default-5b87dddbbf-5h4fs"
Oct 30 22:04:38.542921 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:38.536061    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14ca670b-dd10-4d17-99d0-b1497d64da96-istiod-ca-cert\") pod \"e2e-ait-b3084d7f-openshift-default-5b87dddbbf-5h4fs\" (UID: \"14ca670b-dd10-4d17-99d0-b1497d64da96\") " pod="openshift-ingress/e2e-ait-b3084d7f-openshift-default-5b87dddbbf-5h4fs"
Oct 30 22:04:38.542921 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:38.536125    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14ca670b-dd10-4d17-99d0-b1497d64da96-istio-envoy\") pod \"e2e-ait-b3084d7f-openshift-default-5b87dddbbf-5h4fs\" (UID: \"14ca670b-dd10-4d17-99d0-b1497d64da96\") " pod="openshift-ingress/e2e-ait-b3084d7f-openshift-default-5b87dddbbf-5h4fs"
Oct 30 22:04:38.542921 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:38.537800    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-b3084d7f-openshift-default-dockercfg-xdhdb\""
Oct 30 22:04:38.642594 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:38.638314    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14ca670b-dd10-4d17-99d0-b1497d64da96-workload-certs\") pod \"e2e-ait-b3084d7f-openshift-default-5b87dddbbf-5h4fs\" (UID: \"14ca670b-dd10-4d17-99d0-b1497d64da96\") " pod="openshift-ingress/e2e-ait-b3084d7f-openshift-default-5b87dddbbf-5h4fs"
Oct 30 22:04:38.642594 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:38.638368    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14ca670b-dd10-4d17-99d0-b1497d64da96-istio-data\") pod \"e2e-ait-b3084d7f-openshift-default-5b87dddbbf-5h4fs\" (UID: \"14ca670b-dd10-4d17-99d0-b1497d64da96\") " pod="openshift-ingress/e2e-ait-b3084d7f-openshift-default-5b87dddbbf-5h4fs"
Oct 30 22:04:38.642594 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:38.638433    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14ca670b-dd10-4d17-99d0-b1497d64da96-credential-socket\") pod \"e2e-ait-b3084d7f-openshift-default-5b87dddbbf-5h4fs\" (UID: \"14ca670b-dd10-4d17-99d0-b1497d64da96\") " pod="openshift-ingress/e2e-ait-b3084d7f-openshift-default-5b87dddbbf-5h4fs"
Oct 30 22:04:38.642594 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:38.638460    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14ca670b-dd10-4d17-99d0-b1497d64da96-istiod-ca-cert\") pod \"e2e-ait-b3084d7f-openshift-default-5b87dddbbf-5h4fs\" (UID: \"14ca670b-dd10-4d17-99d0-b1497d64da96\") " pod="openshift-ingress/e2e-ait-b3084d7f-openshift-default-5b87dddbbf-5h4fs"
Oct 30 22:04:38.642594 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:38.638500    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14ca670b-dd10-4d17-99d0-b1497d64da96-istio-envoy\") pod \"e2e-ait-b3084d7f-openshift-default-5b87dddbbf-5h4fs\" (UID: \"14ca670b-dd10-4d17-99d0-b1497d64da96\") " pod="openshift-ingress/e2e-ait-b3084d7f-openshift-default-5b87dddbbf-5h4fs"
Oct 30 22:04:38.642594 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:38.638547    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14ca670b-dd10-4d17-99d0-b1497d64da96-workload-socket\") pod \"e2e-ait-b3084d7f-openshift-default-5b87dddbbf-5h4fs\" (UID: \"14ca670b-dd10-4d17-99d0-b1497d64da96\") " pod="openshift-ingress/e2e-ait-b3084d7f-openshift-default-5b87dddbbf-5h4fs"
Oct 30 22:04:38.642594 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:38.638668    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14ca670b-dd10-4d17-99d0-b1497d64da96-istio-token\") pod \"e2e-ait-b3084d7f-openshift-default-5b87dddbbf-5h4fs\" (UID: \"14ca670b-dd10-4d17-99d0-b1497d64da96\") " pod="openshift-ingress/e2e-ait-b3084d7f-openshift-default-5b87dddbbf-5h4fs"
Oct 30 22:04:38.642594 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:38.638750    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14ca670b-dd10-4d17-99d0-b1497d64da96-istio-podinfo\") pod \"e2e-ait-b3084d7f-openshift-default-5b87dddbbf-5h4fs\" (UID: \"14ca670b-dd10-4d17-99d0-b1497d64da96\") " pod="openshift-ingress/e2e-ait-b3084d7f-openshift-default-5b87dddbbf-5h4fs"
Oct 30 22:04:38.642594 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:38.638791    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bht8\" (UniqueName: \"kubernetes.io/projected/14ca670b-dd10-4d17-99d0-b1497d64da96-kube-api-access-7bht8\") pod \"e2e-ait-b3084d7f-openshift-default-5b87dddbbf-5h4fs\" (UID: \"14ca670b-dd10-4d17-99d0-b1497d64da96\") " pod="openshift-ingress/e2e-ait-b3084d7f-openshift-default-5b87dddbbf-5h4fs"
Oct 30 22:04:38.642594 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:38.639642    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14ca670b-dd10-4d17-99d0-b1497d64da96-workload-certs\") pod \"e2e-ait-b3084d7f-openshift-default-5b87dddbbf-5h4fs\" (UID: \"14ca670b-dd10-4d17-99d0-b1497d64da96\") " pod="openshift-ingress/e2e-ait-b3084d7f-openshift-default-5b87dddbbf-5h4fs"
Oct 30 22:04:38.642594 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:38.639897    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14ca670b-dd10-4d17-99d0-b1497d64da96-istio-data\") pod \"e2e-ait-b3084d7f-openshift-default-5b87dddbbf-5h4fs\" (UID: \"14ca670b-dd10-4d17-99d0-b1497d64da96\") " pod="openshift-ingress/e2e-ait-b3084d7f-openshift-default-5b87dddbbf-5h4fs"
Oct 30 22:04:38.642594 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:38.640117    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14ca670b-dd10-4d17-99d0-b1497d64da96-credential-socket\") pod \"e2e-ait-b3084d7f-openshift-default-5b87dddbbf-5h4fs\" (UID: \"14ca670b-dd10-4d17-99d0-b1497d64da96\") " pod="openshift-ingress/e2e-ait-b3084d7f-openshift-default-5b87dddbbf-5h4fs"
Oct 30 22:04:38.642594 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:38.640829    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14ca670b-dd10-4d17-99d0-b1497d64da96-istiod-ca-cert\") pod \"e2e-ait-b3084d7f-openshift-default-5b87dddbbf-5h4fs\" (UID: \"14ca670b-dd10-4d17-99d0-b1497d64da96\") " pod="openshift-ingress/e2e-ait-b3084d7f-openshift-default-5b87dddbbf-5h4fs"
Oct 30 22:04:38.642594 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:38.641656    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14ca670b-dd10-4d17-99d0-b1497d64da96-workload-socket\") pod \"e2e-ait-b3084d7f-openshift-default-5b87dddbbf-5h4fs\" (UID: \"14ca670b-dd10-4d17-99d0-b1497d64da96\") " pod="openshift-ingress/e2e-ait-b3084d7f-openshift-default-5b87dddbbf-5h4fs"
Oct 30 22:04:38.645232 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:38.645208    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14ca670b-dd10-4d17-99d0-b1497d64da96-istio-envoy\") pod \"e2e-ait-b3084d7f-openshift-default-5b87dddbbf-5h4fs\" (UID: \"14ca670b-dd10-4d17-99d0-b1497d64da96\") " pod="openshift-ingress/e2e-ait-b3084d7f-openshift-default-5b87dddbbf-5h4fs"
Oct 30 22:04:38.678555 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:38.678102    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14ca670b-dd10-4d17-99d0-b1497d64da96-istio-podinfo\") pod \"e2e-ait-b3084d7f-openshift-default-5b87dddbbf-5h4fs\" (UID: \"14ca670b-dd10-4d17-99d0-b1497d64da96\") " pod="openshift-ingress/e2e-ait-b3084d7f-openshift-default-5b87dddbbf-5h4fs"
Oct 30 22:04:38.698336 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:38.696228    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bht8\" (UniqueName: \"kubernetes.io/projected/14ca670b-dd10-4d17-99d0-b1497d64da96-kube-api-access-7bht8\") pod \"e2e-ait-b3084d7f-openshift-default-5b87dddbbf-5h4fs\" (UID: \"14ca670b-dd10-4d17-99d0-b1497d64da96\") " pod="openshift-ingress/e2e-ait-b3084d7f-openshift-default-5b87dddbbf-5h4fs"
Oct 30 22:04:38.703872 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:38.699048    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14ca670b-dd10-4d17-99d0-b1497d64da96-istio-token\") pod \"e2e-ait-b3084d7f-openshift-default-5b87dddbbf-5h4fs\" (UID: \"14ca670b-dd10-4d17-99d0-b1497d64da96\") " pod="openshift-ingress/e2e-ait-b3084d7f-openshift-default-5b87dddbbf-5h4fs"
Oct 30 22:04:38.707400 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:38.706860    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b3084d7f-openshift-default-5b87dddbbf-5h4fs"
Oct 30 22:04:39.957557 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:39.914784    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-bpksk"]
Oct 30 22:04:39.957557 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:39.928706    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-bpksk"]
Oct 30 22:04:39.957557 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:39.928828    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-bpksk"
Oct 30 22:04:40.126295 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:40.027431    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-b3084d7f-openshift-default-5b87dddbbf-5h4fs"]
Oct 30 22:04:40.126295 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:40.046075    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c88c84eb-d49e-49f8-b26e-429659b48ad6-workload-certs\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-bpksk\" (UID: \"c88c84eb-d49e-49f8-b26e-429659b48ad6\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-bpksk"
Oct 30 22:04:40.126295 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:40.046145    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwg2v\" (UniqueName: \"kubernetes.io/projected/c88c84eb-d49e-49f8-b26e-429659b48ad6-kube-api-access-nwg2v\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-bpksk\" (UID: \"c88c84eb-d49e-49f8-b26e-429659b48ad6\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-bpksk"
Oct 30 22:04:40.126295 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:40.046185    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c88c84eb-d49e-49f8-b26e-429659b48ad6-istiod-ca-cert\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-bpksk\" (UID: \"c88c84eb-d49e-49f8-b26e-429659b48ad6\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-bpksk"
Oct 30 22:04:40.126295 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:40.046326    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c88c84eb-d49e-49f8-b26e-429659b48ad6-istio-token\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-bpksk\" (UID: \"c88c84eb-d49e-49f8-b26e-429659b48ad6\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-bpksk"
Oct 30 22:04:40.126295 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:40.046404    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c88c84eb-d49e-49f8-b26e-429659b48ad6-istio-podinfo\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-bpksk\" (UID: \"c88c84eb-d49e-49f8-b26e-429659b48ad6\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-bpksk"
Oct 30 22:04:40.126295 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:40.046471    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c88c84eb-d49e-49f8-b26e-429659b48ad6-workload-socket\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-bpksk\" (UID: \"c88c84eb-d49e-49f8-b26e-429659b48ad6\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-bpksk"
Oct 30 22:04:40.126295 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:40.046591    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c88c84eb-d49e-49f8-b26e-429659b48ad6-istio-envoy\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-bpksk\" (UID: \"c88c84eb-d49e-49f8-b26e-429659b48ad6\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-bpksk"
Oct 30 22:04:40.126295 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:40.046628    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c88c84eb-d49e-49f8-b26e-429659b48ad6-istio-data\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-bpksk\" (UID: \"c88c84eb-d49e-49f8-b26e-429659b48ad6\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-bpksk"
Oct 30 22:04:40.126295 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:40.046705    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c88c84eb-d49e-49f8-b26e-429659b48ad6-credential-socket\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-bpksk\" (UID: \"c88c84eb-d49e-49f8-b26e-429659b48ad6\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-bpksk"
Oct 30 22:04:40.186581 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:40.152670    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c88c84eb-d49e-49f8-b26e-429659b48ad6-istio-token\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-bpksk\" (UID: \"c88c84eb-d49e-49f8-b26e-429659b48ad6\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-bpksk"
Oct 30 22:04:40.186581 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:40.152773    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c88c84eb-d49e-49f8-b26e-429659b48ad6-istio-podinfo\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-bpksk\" (UID: \"c88c84eb-d49e-49f8-b26e-429659b48ad6\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-bpksk"
Oct 30 22:04:40.186581 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:40.152827    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c88c84eb-d49e-49f8-b26e-429659b48ad6-workload-socket\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-bpksk\" (UID: \"c88c84eb-d49e-49f8-b26e-429659b48ad6\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-bpksk"
Oct 30 22:04:40.186581 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:40.152893    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c88c84eb-d49e-49f8-b26e-429659b48ad6-istio-envoy\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-bpksk\" (UID: \"c88c84eb-d49e-49f8-b26e-429659b48ad6\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-bpksk"
Oct 30 22:04:40.186581 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:40.152930    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c88c84eb-d49e-49f8-b26e-429659b48ad6-istio-data\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-bpksk\" (UID: \"c88c84eb-d49e-49f8-b26e-429659b48ad6\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-bpksk"
Oct 30 22:04:40.186581 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:40.152984    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c88c84eb-d49e-49f8-b26e-429659b48ad6-credential-socket\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-bpksk\" (UID: \"c88c84eb-d49e-49f8-b26e-429659b48ad6\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-bpksk"
Oct 30 22:04:40.186581 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:40.153171    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c88c84eb-d49e-49f8-b26e-429659b48ad6-workload-certs\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-bpksk\" (UID: \"c88c84eb-d49e-49f8-b26e-429659b48ad6\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-bpksk"
Oct 30 22:04:40.186581 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:40.153210    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwg2v\" (UniqueName: \"kubernetes.io/projected/c88c84eb-d49e-49f8-b26e-429659b48ad6-kube-api-access-nwg2v\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-bpksk\" (UID: \"c88c84eb-d49e-49f8-b26e-429659b48ad6\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-bpksk"
Oct 30 22:04:40.186581 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:40.153245    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c88c84eb-d49e-49f8-b26e-429659b48ad6-istiod-ca-cert\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-bpksk\" (UID: \"c88c84eb-d49e-49f8-b26e-429659b48ad6\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-bpksk"
Oct 30 22:04:40.203565 ip-10-0-130-60 kubenswrapper[5061]: W1030 22:04:40.200691    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14ca670b_dd10_4d17_99d0_b1497d64da96.slice/crio-dbc4a930f78257258e4bfc7dea69ba0dbba4636566f2b4b1ad0763686f9c9117 WatchSource:0}: Error finding container dbc4a930f78257258e4bfc7dea69ba0dbba4636566f2b4b1ad0763686f9c9117: Status 404 returned error can't find the container with id dbc4a930f78257258e4bfc7dea69ba0dbba4636566f2b4b1ad0763686f9c9117
Oct 30 22:04:40.221478 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:40.221352    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c88c84eb-d49e-49f8-b26e-429659b48ad6-istiod-ca-cert\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-bpksk\" (UID: \"c88c84eb-d49e-49f8-b26e-429659b48ad6\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-bpksk"
Oct 30 22:04:40.222968 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:40.221915    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c88c84eb-d49e-49f8-b26e-429659b48ad6-istio-data\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-bpksk\" (UID: \"c88c84eb-d49e-49f8-b26e-429659b48ad6\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-bpksk"
Oct 30 22:04:40.222968 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:40.222151    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c88c84eb-d49e-49f8-b26e-429659b48ad6-workload-socket\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-bpksk\" (UID: \"c88c84eb-d49e-49f8-b26e-429659b48ad6\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-bpksk"
Oct 30 22:04:40.222968 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:40.222680    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c88c84eb-d49e-49f8-b26e-429659b48ad6-workload-certs\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-bpksk\" (UID: \"c88c84eb-d49e-49f8-b26e-429659b48ad6\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-bpksk"
Oct 30 22:04:40.222968 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:40.222922    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c88c84eb-d49e-49f8-b26e-429659b48ad6-credential-socket\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-bpksk\" (UID: \"c88c84eb-d49e-49f8-b26e-429659b48ad6\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-bpksk"
Oct 30 22:04:40.250296 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:40.248145    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:04:40.250296 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:40.248239    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:04:40.250296 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:40.248284    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:04:40.344550 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:40.337404    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c88c84eb-d49e-49f8-b26e-429659b48ad6-istio-token\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-bpksk\" (UID: \"c88c84eb-d49e-49f8-b26e-429659b48ad6\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-bpksk"
Oct 30 22:04:40.413140 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:40.407517    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c88c84eb-d49e-49f8-b26e-429659b48ad6-istio-podinfo\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-bpksk\" (UID: \"c88c84eb-d49e-49f8-b26e-429659b48ad6\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-bpksk"
Oct 30 22:04:40.627552 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:40.616322    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c88c84eb-d49e-49f8-b26e-429659b48ad6-istio-envoy\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-bpksk\" (UID: \"c88c84eb-d49e-49f8-b26e-429659b48ad6\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-bpksk"
Oct 30 22:04:40.628577 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:40.628494    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwg2v\" (UniqueName: \"kubernetes.io/projected/c88c84eb-d49e-49f8-b26e-429659b48ad6-kube-api-access-nwg2v\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-bpksk\" (UID: \"c88c84eb-d49e-49f8-b26e-429659b48ad6\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-bpksk"
Oct 30 22:04:40.710040 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:40.677919    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-bpksk"
Oct 30 22:04:40.810547 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:40.807908    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b3084d7f-openshift-default-5b87dddbbf-5h4fs" event={"ID":"14ca670b-dd10-4d17-99d0-b1497d64da96","Type":"ContainerStarted","Data":"dbc4a930f78257258e4bfc7dea69ba0dbba4636566f2b4b1ad0763686f9c9117"}
Oct 30 22:04:40.922565 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:40.922074    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-14af5c/distinct-2-w6-14af5c-kserve-58d554bdb6-r5r6j" podUID="990f4e94-708e-45f3-b2ad-01cd50b3257b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:40.960655 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:40.960614    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-17d809/distinct-w4-17d809-kserve-6b8f5c48cd-lhrwp" podUID="ef5e8691-42e1-471b-ba7d-90881028eca7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded"
Oct 30 22:04:41.014650 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:41.014606    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-14af5c/premium-w6-14af5c-kserve-6549647f4-f4gwg" podUID="9b42ee01-c60e-462c-a0e0-b04a52014db6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:41.100553 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:41.055741    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-14af5c/distinct-w6-14af5c-kserve-7c7cb9d69c-hljl9" podUID="1f8d9486-fb2f-491a-a8d7-c23c21bd116a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded"
Oct 30 22:04:41.226792 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:41.224661    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2d1f91/premium-w0-2d1f91-kserve-6965b558cf-59lgw" podUID="fb84233d-25e0-48bc-8899-c3a3cb3cfeb9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:41.299703 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:41.299649    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/distinct-2-w3-f776cb-kserve-7b4d4cc954-d88pf" podUID="5b184062-7fac-43d4-ba5f-21567df85da3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:41.310303 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:41.309616    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2d1f91/simulator-w0-2d1f91-kserve-7699765c74-jwtkb" podUID="44b7d6f6-32d8-4ef4-95bc-eef604fa6b5b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:41.310303 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:41.310008    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-14af5c/simulator-w6-14af5c-kserve-85b8bd96d7-wlcb9" podUID="39766c7d-5f1d-42db-8920-c0321cd12c99" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Oct 30 22:04:41.620551 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:41.599116    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-17d809/simulator-w4-17d809-kserve-78d67b6f4-7hsvb" podUID="e7337cf8-4f59-4afb-9b50-6b6c9fa4a773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Oct 30 22:04:41.620551 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:41.602839    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5be0d1/premium-w2-5be0d1-kserve-759b6fb995-c6gp4" podUID="933f2470-68de-4768-a05b-6c454f16f5f2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded"
Oct 30 22:04:41.620551 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:41.607623    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a07620/simulator-w1-a07620-kserve-55b44674f4-4z9lk" podUID="efe7c063-fb0e-4679-8a7c-13f2a0abaa74" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:42.020551 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:41.977000    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-17d809/distinct-w4-17d809-kserve-6b8f5c48cd-lhrwp" podUID="ef5e8691-42e1-471b-ba7d-90881028eca7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:42.020551 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:41.978018    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-17d809/premium-w4-17d809-kserve-7c9bdbbcbc-5mtmm" podUID="1e0eca81-7db1-482b-8dd6-5b28fb89e010" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Oct 30 22:04:42.020551 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:41.979508    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-17d809/distinct-w4-17d809-kserve-6b8f5c48cd-lhrwp"
Oct 30 22:04:42.132580 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:41.992499    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5be0d1/distinct-w2-5be0d1-kserve-6b85698c8-sxn9k" podUID="7a2dbfc5-908f-41d1-baff-074c56d2cf16" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:42.132580 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:42.009479    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-14af5c/distinct-2-w6-14af5c-kserve-58d554bdb6-r5r6j" podUID="990f4e94-708e-45f3-b2ad-01cd50b3257b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:42.237587 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:42.229938    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2d1f91/premium-w0-2d1f91-kserve-6965b558cf-59lgw" podUID="fb84233d-25e0-48bc-8899-c3a3cb3cfeb9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:42.829822 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:42.602794    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-17d809/simulator-w4-17d809-kserve-78d67b6f4-7hsvb" podUID="e7337cf8-4f59-4afb-9b50-6b6c9fa4a773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:42.829822 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:42.610748    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5be0d1/premium-w2-5be0d1-kserve-759b6fb995-c6gp4" podUID="933f2470-68de-4768-a05b-6c454f16f5f2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:42.829822 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:42.614661    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-14af5c/distinct-2-w6-14af5c-kserve-58d554bdb6-r5r6j"
Oct 30 22:04:42.928549 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:42.910824    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/distinct-2-w3-f776cb-kserve-7b4d4cc954-d88pf" podUID="5b184062-7fac-43d4-ba5f-21567df85da3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:42.928549 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:42.911170    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-14af5c/simulator-w6-14af5c-kserve-85b8bd96d7-wlcb9" podUID="39766c7d-5f1d-42db-8920-c0321cd12c99" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:42.928549 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:42.911418    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-blrrb" podUID="cb4b1ed5-2192-4877-8be7-ca82af831af5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": context deadline exceeded"
Oct 30 22:04:42.928549 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:42.911659    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2d1f91/simulator-w0-2d1f91-kserve-7699765c74-jwtkb" podUID="44b7d6f6-32d8-4ef4-95bc-eef604fa6b5b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 22:04:42.930549 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:42.929903    5061 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-14af5c/simulator-w6-14af5c-kserve-85b8bd96d7-wlcb9" podUID="39766c7d-5f1d-42db-8920-c0321cd12c99" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:42.930549 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:42.930009    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-14af5c/premium-w6-14af5c-kserve-6549647f4-f4gwg" podUID="9b42ee01-c60e-462c-a0e0-b04a52014db6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:42.949205 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:42.932056    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-14af5c/distinct-w6-14af5c-kserve-7c7cb9d69c-hljl9" podUID="1f8d9486-fb2f-491a-a8d7-c23c21bd116a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded"
Oct 30 22:04:42.949205 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:42.610616    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a07620/simulator-w1-a07620-kserve-55b44674f4-4z9lk" podUID="efe7c063-fb0e-4679-8a7c-13f2a0abaa74" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:42.949498 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:42.949456    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2d1f91/premium-w0-2d1f91-kserve-6965b558cf-59lgw"
Oct 30 22:04:42.949822 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:42.949656    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-17d809/simulator-w4-17d809-kserve-78d67b6f4-7hsvb"
Oct 30 22:04:42.978751 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:42.978669    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-17d809/distinct-w4-17d809-kserve-6b8f5c48cd-lhrwp" podUID="ef5e8691-42e1-471b-ba7d-90881028eca7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:43.329463 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:42.732712    5061 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-14af5c/distinct-2-w6-14af5c-kserve-58d554bdb6-r5r6j" podUID="990f4e94-708e-45f3-b2ad-01cd50b3257b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:43.598699 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:43.085694    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5be0d1/distinct-w2-5be0d1-kserve-6b85698c8-sxn9k" podUID="7a2dbfc5-908f-41d1-baff-074c56d2cf16" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 30 22:04:43.598699 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:43.588748    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-4j4xr"]
Oct 30 22:04:43.598699 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:43.096649    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-17d809/premium-w4-17d809-kserve-7c9bdbbcbc-5mtmm" podUID="1e0eca81-7db1-482b-8dd6-5b28fb89e010" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:43.616037 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:43.615148    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2d1f91/premium-w0-2d1f91-kserve-6965b558cf-59lgw" podUID="fb84233d-25e0-48bc-8899-c3a3cb3cfeb9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:43.619549 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:43.105679    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-14af5c/distinct-2-w6-14af5c-kserve-58d554bdb6-r5r6j" podUID="990f4e94-708e-45f3-b2ad-01cd50b3257b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:43.651596 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:43.651412    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-5be0d1/premium-w2-5be0d1-kserve-759b6fb995-c6gp4"
Oct 30 22:04:43.655551 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:43.651950    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f776cb/distinct-2-w3-f776cb-kserve-7b4d4cc954-d88pf"
Oct 30 22:04:43.784765 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:43.651874    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-4j4xr"
Oct 30 22:04:44.106711 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:43.774312    5061 patch_prober.go:29] interesting pod/e2e-ait-b3084d7f-openshift-default-5b87dddbbf-5h4fs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body=
Oct 30 22:04:44.106711 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:43.796617    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b3084d7f-openshift-default-5b87dddbbf-5h4fs" podUID="14ca670b-dd10-4d17-99d0-b1497d64da96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused"
Oct 30 22:04:44.106711 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:43.776406    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df-istio-envoy\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-4j4xr\" (UID: \"aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-4j4xr"
Oct 30 22:04:44.106711 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:43.796715    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wg6m\" (UniqueName: \"kubernetes.io/projected/aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df-kube-api-access-4wg6m\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-4j4xr\" (UID: \"aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-4j4xr"
Oct 30 22:04:44.106711 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:43.796763    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df-istio-token\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-4j4xr\" (UID: \"aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-4j4xr"
Oct 30 22:04:44.106711 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:43.796863    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df-workload-socket\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-4j4xr\" (UID: \"aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-4j4xr"
Oct 30 22:04:44.106711 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:43.796888    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df-workload-certs\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-4j4xr\" (UID: \"aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-4j4xr"
Oct 30 22:04:44.106711 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:43.797176    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df-istio-podinfo\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-4j4xr\" (UID: \"aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-4j4xr"
Oct 30 22:04:44.106711 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:43.797269    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df-credential-socket\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-4j4xr\" (UID: \"aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-4j4xr"
Oct 30 22:04:44.106711 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:43.797345    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df-istiod-ca-cert\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-4j4xr\" (UID: \"aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-4j4xr"
Oct 30 22:04:44.106711 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:43.797413    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df-istio-data\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-4j4xr\" (UID: \"aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-4j4xr"
Oct 30 22:04:44.106711 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:43.775436    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:04:44.106711 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:43.797665    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:04:44.106711 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:43.797721    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:04:44.106711 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:43.837327    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-b3084d7f-openshift-default-5b87dddbbf-5h4fs"
Oct 30 22:04:44.106711 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:43.837397    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-4j4xr"]
Oct 30 22:04:44.106711 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:43.837471    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-14af5c/simulator-w6-14af5c-kserve-85b8bd96d7-wlcb9"
Oct 30 22:04:44.106711 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:43.837576    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2d1f91/simulator-w0-2d1f91-kserve-7699765c74-jwtkb"
Oct 30 22:04:44.106711 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:43.837615    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-14af5c/premium-w6-14af5c-kserve-6549647f4-f4gwg"
Oct 30 22:04:44.106711 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:43.837650    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-14af5c/distinct-w6-14af5c-kserve-7c7cb9d69c-hljl9"
Oct 30 22:04:44.106711 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:43.837686    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a07620/simulator-w1-a07620-kserve-55b44674f4-4z9lk"
Oct 30 22:04:44.106711 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:43.837722    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-5be0d1/distinct-w2-5be0d1-kserve-6b85698c8-sxn9k"
Oct 30 22:04:44.106711 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:43.837756    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-17d809/premium-w4-17d809-kserve-7c9bdbbcbc-5mtmm"
Oct 30 22:04:44.106711 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:43.935746    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-14af5c/distinct-w6-14af5c-kserve-7c7cb9d69c-hljl9" podUID="1f8d9486-fb2f-491a-a8d7-c23c21bd116a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded"
Oct 30 22:04:44.106711 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:43.938618    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a07620/simulator-w1-a07620-kserve-55b44674f4-4z9lk" podUID="efe7c063-fb0e-4679-8a7c-13f2a0abaa74" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:44.106711 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:43.938730    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-14af5c/premium-w6-14af5c-kserve-6549647f4-f4gwg" podUID="9b42ee01-c60e-462c-a0e0-b04a52014db6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:44.106711 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:43.996519    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/distinct-2-w3-f776cb-kserve-7b4d4cc954-d88pf" podUID="5b184062-7fac-43d4-ba5f-21567df85da3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded"
Oct 30 22:04:44.106711 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:43.996957    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-14af5c/simulator-w6-14af5c-kserve-85b8bd96d7-wlcb9" podUID="39766c7d-5f1d-42db-8920-c0321cd12c99" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Oct 30 22:04:44.106711 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:43.997255    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-blrrb" podUID="cb4b1ed5-2192-4877-8be7-ca82af831af5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": context deadline exceeded"
Oct 30 22:04:44.106711 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:44.033956    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2d1f91/simulator-w0-2d1f91-kserve-7699765c74-jwtkb" podUID="44b7d6f6-32d8-4ef4-95bc-eef604fa6b5b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 22:04:44.106711 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:44.034393    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-blrrb"
Oct 30 22:04:44.106711 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:44.045632    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5be0d1/premium-w2-5be0d1-kserve-759b6fb995-c6gp4" podUID="933f2470-68de-4768-a05b-6c454f16f5f2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:44.106711 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:44.047087    5061 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-14af5c/distinct-w6-14af5c-kserve-7c7cb9d69c-hljl9" podUID="1f8d9486-fb2f-491a-a8d7-c23c21bd116a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/health\": context deadline exceeded"
Oct 30 22:04:44.106711 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:44.047154    5061 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-14af5c/premium-w6-14af5c-kserve-6549647f4-f4gwg" podUID="9b42ee01-c60e-462c-a0e0-b04a52014db6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": context deadline exceeded"
Oct 30 22:04:44.106711 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:44.047191    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-17d809/simulator-w4-17d809-kserve-78d67b6f4-7hsvb" podUID="e7337cf8-4f59-4afb-9b50-6b6c9fa4a773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Oct 30 22:04:44.212557 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:44.125134    5061 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-5be0d1/distinct-w2-5be0d1-kserve-6b85698c8-sxn9k" podUID="7a2dbfc5-908f-41d1-baff-074c56d2cf16" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": context deadline exceeded"
Oct 30 22:04:44.316555 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:44.276110    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df-istio-token\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-4j4xr\" (UID: \"aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-4j4xr"
Oct 30 22:04:44.316555 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:44.276227    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df-workload-socket\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-4j4xr\" (UID: \"aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-4j4xr"
Oct 30 22:04:44.316555 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:44.276263    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df-workload-certs\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-4j4xr\" (UID: \"aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-4j4xr"
Oct 30 22:04:44.374622 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:44.347221    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-17d809/distinct-w4-17d809-kserve-6b8f5c48cd-lhrwp" podUID="ef5e8691-42e1-471b-ba7d-90881028eca7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded"
Oct 30 22:04:44.407561 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:44.393173    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df-istio-podinfo\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-4j4xr\" (UID: \"aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-4j4xr"
Oct 30 22:04:44.407561 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:44.393327    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df-credential-socket\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-4j4xr\" (UID: \"aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-4j4xr"
Oct 30 22:04:44.407561 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:44.393423    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df-istiod-ca-cert\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-4j4xr\" (UID: \"aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-4j4xr"
Oct 30 22:04:44.407561 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:44.393504    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df-istio-data\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-4j4xr\" (UID: \"aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-4j4xr"
Oct 30 22:04:44.407561 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:44.397630    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df-credential-socket\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-4j4xr\" (UID: \"aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-4j4xr"
Oct 30 22:04:44.513562 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:44.409780    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df-workload-socket\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-4j4xr\" (UID: \"aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-4j4xr"
Oct 30 22:04:44.513562 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:44.410039    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df-workload-certs\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-4j4xr\" (UID: \"aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-4j4xr"
Oct 30 22:04:44.513562 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:44.429370    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df-istio-data\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-4j4xr\" (UID: \"aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-4j4xr"
Oct 30 22:04:44.582552 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:44.556750    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df-istio-envoy\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-4j4xr\" (UID: \"aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-4j4xr"
Oct 30 22:04:44.582552 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:44.556821    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wg6m\" (UniqueName: \"kubernetes.io/projected/aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df-kube-api-access-4wg6m\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-4j4xr\" (UID: \"aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-4j4xr"
Oct 30 22:04:44.654354 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:44.630625    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-14af5c/distinct-2-w6-14af5c-kserve-58d554bdb6-r5r6j" podUID="990f4e94-708e-45f3-b2ad-01cd50b3257b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded"
Oct 30 22:04:44.654354 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:44.653595    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2d1f91/premium-w0-2d1f91-kserve-6965b558cf-59lgw" podUID="fb84233d-25e0-48bc-8899-c3a3cb3cfeb9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 30 22:04:44.770563 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:44.750284    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5be0d1/distinct-w2-5be0d1-kserve-6b85698c8-sxn9k" podUID="7a2dbfc5-908f-41d1-baff-074c56d2cf16" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 30 22:04:44.817394 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:44.797252    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-17d809/premium-w4-17d809-kserve-7c9bdbbcbc-5mtmm" podUID="1e0eca81-7db1-482b-8dd6-5b28fb89e010" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Oct 30 22:04:44.845554 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:44.779707    5061 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-17d809/distinct-w4-17d809-kserve-6b8f5c48cd-lhrwp" podUID="ef5e8691-42e1-471b-ba7d-90881028eca7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:44.897555 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:44.872119    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df-istiod-ca-cert\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-4j4xr\" (UID: \"aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-4j4xr"
Oct 30 22:04:45.183557 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:44.946256    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df-istio-podinfo\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-4j4xr\" (UID: \"aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-4j4xr"
Oct 30 22:04:45.349723 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:44.947222    5061 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-f776cb/distinct-2-w3-f776cb-kserve-7b4d4cc954-d88pf" podUID="5b184062-7fac-43d4-ba5f-21567df85da3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/health\": context deadline exceeded"
Oct 30 22:04:45.349986 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:44.947299    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a07620/simulator-w1-a07620-kserve-55b44674f4-4z9lk" podUID="efe7c063-fb0e-4679-8a7c-13f2a0abaa74" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Oct 30 22:04:45.488607 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:44.947323    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-14af5c/premium-w6-14af5c-kserve-6549647f4-f4gwg" podUID="9b42ee01-c60e-462c-a0e0-b04a52014db6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded"
Oct 30 22:04:45.574575 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:44.948474    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-14af5c/distinct-w6-14af5c-kserve-7c7cb9d69c-hljl9" podUID="1f8d9486-fb2f-491a-a8d7-c23c21bd116a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:45.574575 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:44.965154    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df-istio-token\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-4j4xr\" (UID: \"aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-4j4xr"
Oct 30 22:04:45.574575 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:45.094323    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df-istio-envoy\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-4j4xr\" (UID: \"aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-4j4xr"
Oct 30 22:04:45.574575 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:45.132888    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5be0d1/premium-w2-5be0d1-kserve-759b6fb995-c6gp4" podUID="933f2470-68de-4768-a05b-6c454f16f5f2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded"
Oct 30 22:04:45.574575 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:45.132937    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-17d809/simulator-w4-17d809-kserve-78d67b6f4-7hsvb" podUID="e7337cf8-4f59-4afb-9b50-6b6c9fa4a773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Oct 30 22:04:45.574575 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:45.133625    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/distinct-2-w3-f776cb-kserve-7b4d4cc954-d88pf" podUID="5b184062-7fac-43d4-ba5f-21567df85da3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:45.574575 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:45.133657    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-14af5c/simulator-w6-14af5c-kserve-85b8bd96d7-wlcb9" podUID="39766c7d-5f1d-42db-8920-c0321cd12c99" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:45.574575 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:45.133679    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-blrrb" podUID="cb4b1ed5-2192-4877-8be7-ca82af831af5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": context deadline exceeded"
Oct 30 22:04:45.574575 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:45.133700    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2d1f91/simulator-w0-2d1f91-kserve-7699765c74-jwtkb" podUID="44b7d6f6-32d8-4ef4-95bc-eef604fa6b5b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 22:04:45.574575 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:45.296339    5061 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-2d1f91/premium-w0-2d1f91-kserve-6965b558cf-59lgw" podUID="fb84233d-25e0-48bc-8899-c3a3cb3cfeb9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": context deadline exceeded"
Oct 30 22:04:45.574575 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:45.502684    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-17d809/distinct-w4-17d809-kserve-6b8f5c48cd-lhrwp" podUID="ef5e8691-42e1-471b-ba7d-90881028eca7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded"
Oct 30 22:04:45.574575 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:45.548070    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-b3084d7f-openshift-default-5b87dddbbf-5h4fs"
Oct 30 22:04:45.574575 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:45.548185    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-bpksk"]
Oct 30 22:04:45.574575 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:45.474520    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wg6m\" (UniqueName: \"kubernetes.io/projected/aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df-kube-api-access-4wg6m\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-4j4xr\" (UID: \"aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-4j4xr"
Oct 30 22:04:45.608938 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:45.607274    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-b3084d7f-openshift-default-5b87dddbbf-5h4fs" podStartSLOduration=7.607255264 podStartE2EDuration="7.607255264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:43.713002076 +0000 UTC m=+1337.325771834" watchObservedRunningTime="2026-10-30 22:04:45.607255264 +0000 UTC m=+1339.220025025"
Oct 30 22:04:45.663550 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:45.634790    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-14af5c/distinct-2-w6-14af5c-kserve-58d554bdb6-r5r6j" podUID="990f4e94-708e-45f3-b2ad-01cd50b3257b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:45.663550 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:45.658005    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2d1f91/premium-w0-2d1f91-kserve-6965b558cf-59lgw" podUID="fb84233d-25e0-48bc-8899-c3a3cb3cfeb9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 30 22:04:45.774319 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:45.774278    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-bpksk" event={"ID":"c88c84eb-d49e-49f8-b26e-429659b48ad6","Type":"ContainerStarted","Data":"2a6734158b36a1ecf8da88cb7db5caf2d20c24af9c14631ab97c48c67b898c74"}
Oct 30 22:04:45.838895 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:45.838809    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-4j4xr"
Oct 30 22:04:45.953218 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:45.910426    5061 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-5be0d1/premium-w2-5be0d1-kserve-759b6fb995-c6gp4" podUID="933f2470-68de-4768-a05b-6c454f16f5f2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:45.953218 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:45.896394    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-17d809/premium-w4-17d809-kserve-7c9bdbbcbc-5mtmm" podUID="1e0eca81-7db1-482b-8dd6-5b28fb89e010" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Oct 30 22:04:45.953218 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:45.939648    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5be0d1/distinct-w2-5be0d1-kserve-6b85698c8-sxn9k" podUID="7a2dbfc5-908f-41d1-baff-074c56d2cf16" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 30 22:04:45.966194 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:45.966155    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b3084d7f-openshift-default-5b87dddbbf-5h4fs" event={"ID":"14ca670b-dd10-4d17-99d0-b1497d64da96","Type":"ContainerStarted","Data":"e265965f3aa420f19497d908d456f7ab0c9b417c759ff5030ffe22eb68186dad"}
Oct 30 22:04:45.982549 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:45.970345    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-b3084d7f-openshift-default-5b87dddbbf-5h4fs"
Oct 30 22:04:46.043862 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:46.043635    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-b3084d7f-openshift-default-5b87dddbbf-5h4fs"
Oct 30 22:04:46.044674 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:46.044649    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-5be0d1/distinct-w2-5be0d1-kserve-6b85698c8-sxn9k"
Oct 30 22:04:46.116819 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:46.116489    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-14af5c/simulator-w6-14af5c-kserve-85b8bd96d7-wlcb9"
Oct 30 22:04:46.167291 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:46.147748    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2d1f91/premium-w0-2d1f91-kserve-6965b558cf-59lgw"
Oct 30 22:04:46.167291 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:46.148710    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-14af5c/distinct-2-w6-14af5c-kserve-58d554bdb6-r5r6j"
Oct 30 22:04:46.167291 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:46.149896    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-blrrb"
Oct 30 22:04:46.167291 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:46.158380    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-5be0d1/premium-w2-5be0d1-kserve-759b6fb995-c6gp4"
Oct 30 22:04:46.167291 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:46.160913    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a07620/simulator-w1-a07620-kserve-55b44674f4-4z9lk"
Oct 30 22:04:46.232673 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:46.232636    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-17d809/simulator-w4-17d809-kserve-78d67b6f4-7hsvb"
Oct 30 22:04:46.232974 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:46.232960    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-17d809/premium-w4-17d809-kserve-7c9bdbbcbc-5mtmm"
Oct 30 22:04:46.233147 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:46.233135    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-14af5c/distinct-w6-14af5c-kserve-7c7cb9d69c-hljl9"
Oct 30 22:04:46.233281 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:46.233271    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2d1f91/simulator-w0-2d1f91-kserve-7699765c74-jwtkb"
Oct 30 22:04:46.238788 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:46.238765    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f776cb/distinct-2-w3-f776cb-kserve-7b4d4cc954-d88pf"
Oct 30 22:04:46.239101 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:46.239052    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-14af5c/premium-w6-14af5c-kserve-6549647f4-f4gwg"
Oct 30 22:04:46.243201 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:46.241222    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-17d809/distinct-w4-17d809-kserve-6b8f5c48cd-lhrwp"
Oct 30 22:04:46.297354 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:46.297320    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c586c"]
Oct 30 22:04:46.303790 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:46.303763    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c586c"
Oct 30 22:04:46.312983 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:46.312914    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c586c"]
Oct 30 22:04:46.357949 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:46.357915    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/dc212ddd-b71e-44c3-88f2-f7c0936bf7d4-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c586c\" (UID: \"dc212ddd-b71e-44c3-88f2-f7c0936bf7d4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c586c"
Oct 30 22:04:46.358106 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:46.357992    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86tlw\" (UniqueName: \"kubernetes.io/projected/dc212ddd-b71e-44c3-88f2-f7c0936bf7d4-kube-api-access-86tlw\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c586c\" (UID: \"dc212ddd-b71e-44c3-88f2-f7c0936bf7d4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c586c"
Oct 30 22:04:46.358106 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:46.358025    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/dc212ddd-b71e-44c3-88f2-f7c0936bf7d4-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c586c\" (UID: \"dc212ddd-b71e-44c3-88f2-f7c0936bf7d4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c586c"
Oct 30 22:04:46.358106 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:46.358050    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/dc212ddd-b71e-44c3-88f2-f7c0936bf7d4-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c586c\" (UID: \"dc212ddd-b71e-44c3-88f2-f7c0936bf7d4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c586c"
Oct 30 22:04:46.358281 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:46.358122    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc212ddd-b71e-44c3-88f2-f7c0936bf7d4-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c586c\" (UID: \"dc212ddd-b71e-44c3-88f2-f7c0936bf7d4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c586c"
Oct 30 22:04:46.358281 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:46.358164    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc212ddd-b71e-44c3-88f2-f7c0936bf7d4-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c586c\" (UID: \"dc212ddd-b71e-44c3-88f2-f7c0936bf7d4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c586c"
Oct 30 22:04:46.358462 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:46.358442    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/dc212ddd-b71e-44c3-88f2-f7c0936bf7d4-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c586c\" (UID: \"dc212ddd-b71e-44c3-88f2-f7c0936bf7d4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c586c"
Oct 30 22:04:46.358635 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:46.358617    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7185-215e51"]
Oct 30 22:04:46.358818 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:46.358794    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7185-215e51" podUID="cdd4b251-9868-4b9c-ad82-8c913536950b" containerName="test-curl-7185-215e51" containerID="cri-o://3e196b6b24093386aef9c49ab1fec521432446176581c5dace607c76821b4e2c" gracePeriod=2
Oct 30 22:04:46.378314 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:46.360887    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 30 22:04:46.378314 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:46.360996    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-dwlzg\""
Oct 30 22:04:46.378314 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:46.361040    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 30 22:04:46.378314 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:46.365812    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7185-215e51"]
Oct 30 22:04:46.378314 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:46.365910    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 30 22:04:46.460593 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:46.460555    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/dc212ddd-b71e-44c3-88f2-f7c0936bf7d4-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c586c\" (UID: \"dc212ddd-b71e-44c3-88f2-f7c0936bf7d4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c586c"
Oct 30 22:04:46.460777 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:46.460649    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/dc212ddd-b71e-44c3-88f2-f7c0936bf7d4-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c586c\" (UID: \"dc212ddd-b71e-44c3-88f2-f7c0936bf7d4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c586c"
Oct 30 22:04:46.460777 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:46.460707    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86tlw\" (UniqueName: \"kubernetes.io/projected/dc212ddd-b71e-44c3-88f2-f7c0936bf7d4-kube-api-access-86tlw\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c586c\" (UID: \"dc212ddd-b71e-44c3-88f2-f7c0936bf7d4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c586c"
Oct 30 22:04:46.460900 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:46.460774    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/dc212ddd-b71e-44c3-88f2-f7c0936bf7d4-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c586c\" (UID: \"dc212ddd-b71e-44c3-88f2-f7c0936bf7d4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c586c"
Oct 30 22:04:46.460900 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:46.460802    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/dc212ddd-b71e-44c3-88f2-f7c0936bf7d4-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c586c\" (UID: \"dc212ddd-b71e-44c3-88f2-f7c0936bf7d4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c586c"
Oct 30 22:04:46.460900 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:46.460875    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc212ddd-b71e-44c3-88f2-f7c0936bf7d4-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c586c\" (UID: \"dc212ddd-b71e-44c3-88f2-f7c0936bf7d4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c586c"
Oct 30 22:04:46.461067 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:46.460911    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc212ddd-b71e-44c3-88f2-f7c0936bf7d4-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c586c\" (UID: \"dc212ddd-b71e-44c3-88f2-f7c0936bf7d4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c586c"
Oct 30 22:04:46.461430 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:46.461401    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/dc212ddd-b71e-44c3-88f2-f7c0936bf7d4-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c586c\" (UID: \"dc212ddd-b71e-44c3-88f2-f7c0936bf7d4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c586c"
Oct 30 22:04:46.464095 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:46.464069    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/dc212ddd-b71e-44c3-88f2-f7c0936bf7d4-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c586c\" (UID: \"dc212ddd-b71e-44c3-88f2-f7c0936bf7d4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c586c"
Oct 30 22:04:46.465173 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:46.465147    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/dc212ddd-b71e-44c3-88f2-f7c0936bf7d4-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c586c\" (UID: \"dc212ddd-b71e-44c3-88f2-f7c0936bf7d4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c586c"
Oct 30 22:04:46.465275 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:46.465228    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/dc212ddd-b71e-44c3-88f2-f7c0936bf7d4-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c586c\" (UID: \"dc212ddd-b71e-44c3-88f2-f7c0936bf7d4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c586c"
Oct 30 22:04:46.466794 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:46.466770    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc212ddd-b71e-44c3-88f2-f7c0936bf7d4-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c586c\" (UID: \"dc212ddd-b71e-44c3-88f2-f7c0936bf7d4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c586c"
Oct 30 22:04:46.516018 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:46.515980    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86tlw\" (UniqueName: \"kubernetes.io/projected/dc212ddd-b71e-44c3-88f2-f7c0936bf7d4-kube-api-access-86tlw\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c586c\" (UID: \"dc212ddd-b71e-44c3-88f2-f7c0936bf7d4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c586c"
Oct 30 22:04:46.516874 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:46.516850    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc212ddd-b71e-44c3-88f2-f7c0936bf7d4-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c586c\" (UID: \"dc212ddd-b71e-44c3-88f2-f7c0936bf7d4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c586c"
Oct 30 22:04:46.659646 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:46.659562    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c586c"
Oct 30 22:04:46.717216 ip-10-0-130-60 kubenswrapper[5061]: W1030 22:04:46.717181    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa7b6c04_24c2_40b5_8d2a_d0c4da6f41df.slice/crio-8b31f107fa355aad889b32b9f53a79d5f7647f23618b5498c67cef86a38595e6 WatchSource:0}: Error finding container 8b31f107fa355aad889b32b9f53a79d5f7647f23618b5498c67cef86a38595e6: Status 404 returned error can't find the container with id 8b31f107fa355aad889b32b9f53a79d5f7647f23618b5498c67cef86a38595e6
Oct 30 22:04:46.720586 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:46.720170    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-4j4xr"]
Oct 30 22:04:46.720710 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:46.720649    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:04:46.720782 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:46.720740    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:04:46.720836 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:46.720782    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:04:46.971976 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:46.971898    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-4j4xr" event={"ID":"aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df","Type":"ContainerStarted","Data":"8b977b2c9b1fc9c6ae43d67d2948b12ffc7573a3cf71688b8bc9de22f1bf0fea"}
Oct 30 22:04:46.971976 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:46.971960    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-4j4xr" event={"ID":"aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df","Type":"ContainerStarted","Data":"8b31f107fa355aad889b32b9f53a79d5f7647f23618b5498c67cef86a38595e6"}
Oct 30 22:04:46.973667 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:46.973635    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-bpksk" event={"ID":"c88c84eb-d49e-49f8-b26e-429659b48ad6","Type":"ContainerStarted","Data":"e99ff9345b2bd8c8ff797cf09541c95857c43bb9d33cbf9c830df59f0b23e2fc"}
Oct 30 22:04:47.064444 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:47.064399    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c586c"]
Oct 30 22:04:47.091489 ip-10-0-130-60 kubenswrapper[5061]: W1030 22:04:47.091453    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc212ddd_b71e_44c3_88f2_f7c0936bf7d4.slice/crio-1480db6712e38db03efeb5a5a8a01146be165ef5e7519849301c790993b59636 WatchSource:0}: Error finding container 1480db6712e38db03efeb5a5a8a01146be165ef5e7519849301c790993b59636: Status 404 returned error can't find the container with id 1480db6712e38db03efeb5a5a8a01146be165ef5e7519849301c790993b59636
Oct 30 22:04:47.678177 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:47.678145    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-bpksk"
Oct 30 22:04:47.679493 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:47.679460    5061 patch_prober.go:29] interesting pod/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-bpksk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Oct 30 22:04:47.679631 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:47.679510    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-bpksk" podUID="c88c84eb-d49e-49f8-b26e-429659b48ad6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Oct 30 22:04:47.906197 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:47.906163    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-4j4xr"
Oct 30 22:04:47.915550 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:47.907511    5061 patch_prober.go:29] interesting pod/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-4j4xr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body=
Oct 30 22:04:47.915550 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:47.907584    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-4j4xr" podUID="aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused"
Oct 30 22:04:47.999363 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:47.996598    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c586c" event={"ID":"dc212ddd-b71e-44c3-88f2-f7c0936bf7d4","Type":"ContainerStarted","Data":"1480db6712e38db03efeb5a5a8a01146be165ef5e7519849301c790993b59636"}
Oct 30 22:04:48.070610 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:48.070572    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37cbad61-v9npg"]
Oct 30 22:04:48.072273 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:48.071616    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-bpksk" podStartSLOduration=9.071600021 podStartE2EDuration="9.071600021s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:48.06965527 +0000 UTC m=+1341.682425040" watchObservedRunningTime="2026-10-30 22:04:48.071600021 +0000 UTC m=+1341.684369781"
Oct 30 22:04:48.072273 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:48.071673    5061 cpu_manager.go:414] "containerMap: removing container" podUID="cdd4b251-9868-4b9c-ad82-8c913536950b" containerName="test-curl-7185-215e51"
Oct 30 22:04:48.072273 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:48.071695    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdd4b251-9868-4b9c-ad82-8c913536950b" containerName="test-curl-7185-215e51"
Oct 30 22:04:48.072273 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:48.071844    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdd4b251-9868-4b9c-ad82-8c913536950b" containerName="test-curl-7185-215e51"
Oct 30 22:04:48.089147 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:48.086902    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37cbad61-v9npg"
Oct 30 22:04:48.140562 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:48.139910    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gdmfg\""
Oct 30 22:04:48.145663 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:48.140900    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:04:48.161812 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:48.160280    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37cbad61-v9npg"]
Oct 30 22:04:48.272285 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:48.271988    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d82b81f-d252-4fd6-ac51-bc4dccbce83f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37cbad61-v9npg\" (UID: \"1d82b81f-d252-4fd6-ac51-bc4dccbce83f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37cbad61-v9npg"
Oct 30 22:04:48.272285 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:48.272128    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zljt6\" (UniqueName: \"kubernetes.io/projected/1d82b81f-d252-4fd6-ac51-bc4dccbce83f-kube-api-access-zljt6\") pod \"maas-api-revoke-sub-37cbad61-v9npg\" (UID: \"1d82b81f-d252-4fd6-ac51-bc4dccbce83f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37cbad61-v9npg"
Oct 30 22:04:48.272622 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:48.272258    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-4j4xr" podStartSLOduration=7.272226875 podStartE2EDuration="7.272226875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:48.231021249 +0000 UTC m=+1341.843791019" watchObservedRunningTime="2026-10-30 22:04:48.272226875 +0000 UTC m=+1341.884996634"
Oct 30 22:04:48.272622 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:48.272465    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d82b81f-d252-4fd6-ac51-bc4dccbce83f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37cbad61-v9npg\" (UID: \"1d82b81f-d252-4fd6-ac51-bc4dccbce83f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37cbad61-v9npg"
Oct 30 22:04:48.374518 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:48.374486    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d82b81f-d252-4fd6-ac51-bc4dccbce83f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37cbad61-v9npg\" (UID: \"1d82b81f-d252-4fd6-ac51-bc4dccbce83f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37cbad61-v9npg"
Oct 30 22:04:48.374676 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:48.374655    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zljt6\" (UniqueName: \"kubernetes.io/projected/1d82b81f-d252-4fd6-ac51-bc4dccbce83f-kube-api-access-zljt6\") pod \"maas-api-revoke-sub-37cbad61-v9npg\" (UID: \"1d82b81f-d252-4fd6-ac51-bc4dccbce83f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37cbad61-v9npg"
Oct 30 22:04:48.374973 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:48.374958    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d82b81f-d252-4fd6-ac51-bc4dccbce83f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37cbad61-v9npg\" (UID: \"1d82b81f-d252-4fd6-ac51-bc4dccbce83f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37cbad61-v9npg"
Oct 30 22:04:48.375798 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:48.375775    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d82b81f-d252-4fd6-ac51-bc4dccbce83f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37cbad61-v9npg\" (UID: \"1d82b81f-d252-4fd6-ac51-bc4dccbce83f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37cbad61-v9npg"
Oct 30 22:04:48.379319 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:48.379293    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d82b81f-d252-4fd6-ac51-bc4dccbce83f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37cbad61-v9npg\" (UID: \"1d82b81f-d252-4fd6-ac51-bc4dccbce83f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37cbad61-v9npg"
Oct 30 22:04:48.467131 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:48.467076    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zljt6\" (UniqueName: \"kubernetes.io/projected/1d82b81f-d252-4fd6-ac51-bc4dccbce83f-kube-api-access-zljt6\") pod \"maas-api-revoke-sub-37cbad61-v9npg\" (UID: \"1d82b81f-d252-4fd6-ac51-bc4dccbce83f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37cbad61-v9npg"
Oct 30 22:04:48.678628 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:48.678590    5061 patch_prober.go:29] interesting pod/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-bpksk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Oct 30 22:04:48.679129 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:48.678651    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-bpksk" podUID="c88c84eb-d49e-49f8-b26e-429659b48ad6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Oct 30 22:04:48.751378 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:48.751345    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37cbad61-v9npg"
Oct 30 22:04:48.881870 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:48.881837    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7185-215e51"
Oct 30 22:04:48.906742 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:48.906294    5061 patch_prober.go:29] interesting pod/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-4j4xr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body=
Oct 30 22:04:48.906742 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:48.906353    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-4j4xr" podUID="aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused"
Oct 30 22:04:48.936607 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:04:48.930838    5061 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7185-215e51\" is forbidden: User \"system:node:ip-10-0-130-60.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-60.us-west-2.compute.internal' and this object" podUID="cdd4b251-9868-4b9c-ad82-8c913536950b" pod="opendatahub/test-curl-7185-215e51"
Oct 30 22:04:48.985089 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:48.985047    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdd4b251-9868-4b9c-ad82-8c913536950b-kube-api-access-lfq8d\" (UniqueName: \"kubernetes.io/projected/cdd4b251-9868-4b9c-ad82-8c913536950b-kube-api-access-lfq8d\") pod \"cdd4b251-9868-4b9c-ad82-8c913536950b\" (UID: \"cdd4b251-9868-4b9c-ad82-8c913536950b\") "
Oct 30 22:04:48.987769 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:48.987743    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdd4b251-9868-4b9c-ad82-8c913536950b-kube-api-access-lfq8d" pod "cdd4b251-9868-4b9c-ad82-8c913536950b" (UID: "cdd4b251-9868-4b9c-ad82-8c913536950b"). InnerVolumeSpecName "kube-api-access-lfq8d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:04:49.048907 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:49.007399    5061 generic.go:356] "Generic (PLEG): container finished" podID="cdd4b251-9868-4b9c-ad82-8c913536950b" containerID="3e196b6b24093386aef9c49ab1fec521432446176581c5dace607c76821b4e2c" exitCode=137
Oct 30 22:04:49.048907 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:49.008725    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7185-215e51"
Oct 30 22:04:49.048907 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:49.029488    5061 scope.go:122] "RemoveContainer" containerID="3e196b6b24093386aef9c49ab1fec521432446176581c5dace607c76821b4e2c"
Oct 30 22:04:49.074956 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:49.074935    5061 scope.go:122] "RemoveContainer" containerID="3e196b6b24093386aef9c49ab1fec521432446176581c5dace607c76821b4e2c"
Oct 30 22:04:49.075232 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:04:49.075212    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e196b6b24093386aef9c49ab1fec521432446176581c5dace607c76821b4e2c\": container with ID starting with 3e196b6b24093386aef9c49ab1fec521432446176581c5dace607c76821b4e2c not found: ID does not exist" containerID="3e196b6b24093386aef9c49ab1fec521432446176581c5dace607c76821b4e2c"
Oct 30 22:04:49.075268 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:49.075245    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e196b6b24093386aef9c49ab1fec521432446176581c5dace607c76821b4e2c"} err="failed to get container status \"3e196b6b24093386aef9c49ab1fec521432446176581c5dace607c76821b4e2c\": rpc error: code = NotFound desc = could not find container \"3e196b6b24093386aef9c49ab1fec521432446176581c5dace607c76821b4e2c\": container with ID starting with 3e196b6b24093386aef9c49ab1fec521432446176581c5dace607c76821b4e2c not found: ID does not exist"
Oct 30 22:04:49.079072 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:04:49.079043    5061 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7185-215e51\" is forbidden: User \"system:node:ip-10-0-130-60.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-60.us-west-2.compute.internal' and this object" podUID="cdd4b251-9868-4b9c-ad82-8c913536950b" pod="opendatahub/test-curl-7185-215e51"
Oct 30 22:04:49.087427 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:49.087399    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfq8d\" (UniqueName: \"kubernetes.io/projected/cdd4b251-9868-4b9c-ad82-8c913536950b-kube-api-access-lfq8d\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:04:49.151818 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:04:49.148789    5061 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7185-215e51\" is forbidden: User \"system:node:ip-10-0-130-60.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-60.us-west-2.compute.internal' and this object" podUID="cdd4b251-9868-4b9c-ad82-8c913536950b" pod="opendatahub/test-curl-7185-215e51"
Oct 30 22:04:49.199027 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:49.198994    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37cbad61-v9npg"]
Oct 30 22:04:49.199483 ip-10-0-130-60 kubenswrapper[5061]: W1030 22:04:49.199452    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d82b81f_d252_4fd6_ac51_bc4dccbce83f.slice/crio-e28cfa386fe76f1201f93b6b917b69160040c1d53544546c7b5597a7494d8783 WatchSource:0}: Error finding container e28cfa386fe76f1201f93b6b917b69160040c1d53544546c7b5597a7494d8783: Status 404 returned error can't find the container with id e28cfa386fe76f1201f93b6b917b69160040c1d53544546c7b5597a7494d8783
Oct 30 22:04:49.678775 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:49.678737    5061 patch_prober.go:29] interesting pod/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-bpksk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Oct 30 22:04:49.679189 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:49.678792    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-bpksk" podUID="c88c84eb-d49e-49f8-b26e-429659b48ad6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Oct 30 22:04:49.916103 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:49.915661    5061 patch_prober.go:29] interesting pod/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-4j4xr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body=
Oct 30 22:04:49.916103 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:49.915721    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-4j4xr" podUID="aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused"
Oct 30 22:04:50.077567 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:50.072713    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37cbad61-v9npg" event={"ID":"1d82b81f-d252-4fd6-ac51-bc4dccbce83f","Type":"ContainerStarted","Data":"93ceb97a68e08e3dad5e14310f54634b405fdf465e0d5a27d43ffb00d18735ca"}
Oct 30 22:04:50.077567 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:50.072762    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37cbad61-v9npg" event={"ID":"1d82b81f-d252-4fd6-ac51-bc4dccbce83f","Type":"ContainerStarted","Data":"e28cfa386fe76f1201f93b6b917b69160040c1d53544546c7b5597a7494d8783"}
Oct 30 22:04:50.132548 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:04:50.121740    5061 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7185-215e51\" is forbidden: User \"system:node:ip-10-0-130-60.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-60.us-west-2.compute.internal' and this object" podUID="cdd4b251-9868-4b9c-ad82-8c913536950b" pod="opendatahub/test-curl-7185-215e51"
Oct 30 22:04:50.271082 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:50.271023    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37cbad61-v9npg" podStartSLOduration=2.271006268 podStartE2EDuration="2.271006268s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:50.242957695 +0000 UTC m=+1343.855727482" watchObservedRunningTime="2026-10-30 22:04:50.271006268 +0000 UTC m=+1343.883776026"
Oct 30 22:04:50.576730 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:50.576695    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cdd4b251-9868-4b9c-ad82-8c913536950b" path="/var/lib/kubelet/pods/cdd4b251-9868-4b9c-ad82-8c913536950b/volumes"
Oct 30 22:04:50.679174 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:50.679142    5061 patch_prober.go:29] interesting pod/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-bpksk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Oct 30 22:04:50.679592 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:50.679203    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-bpksk" podUID="c88c84eb-d49e-49f8-b26e-429659b48ad6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Oct 30 22:04:50.907118 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:50.907086    5061 patch_prober.go:29] interesting pod/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-4j4xr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body=
Oct 30 22:04:50.907287 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:50.907146    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-4j4xr" podUID="aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused"
Oct 30 22:04:51.086821 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:51.086785    5061 generic.go:356] "Generic (PLEG): container finished" podID="1d82b81f-d252-4fd6-ac51-bc4dccbce83f" containerID="93ceb97a68e08e3dad5e14310f54634b405fdf465e0d5a27d43ffb00d18735ca" exitCode=0
Oct 30 22:04:51.087010 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:51.086938    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37cbad61-v9npg" event={"ID":"1d82b81f-d252-4fd6-ac51-bc4dccbce83f","Type":"ContainerDied","Data":"93ceb97a68e08e3dad5e14310f54634b405fdf465e0d5a27d43ffb00d18735ca"}
Oct 30 22:04:51.683566 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:51.680846    5061 patch_prober.go:29] interesting pod/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-bpksk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Oct 30 22:04:51.683566 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:51.680913    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-bpksk" podUID="c88c84eb-d49e-49f8-b26e-429659b48ad6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Oct 30 22:04:51.906377 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:51.906338    5061 patch_prober.go:29] interesting pod/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-4j4xr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body=
Oct 30 22:04:51.906569 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:51.906407    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-4j4xr" podUID="aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused"
Oct 30 22:04:52.661097 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:52.660788    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37cbad61-v9npg"
Oct 30 22:04:52.679463 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:52.679425    5061 patch_prober.go:29] interesting pod/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-bpksk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Oct 30 22:04:52.679653 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:52.679475    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-bpksk" podUID="c88c84eb-d49e-49f8-b26e-429659b48ad6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Oct 30 22:04:52.682430 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:52.682009    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d82b81f-d252-4fd6-ac51-bc4dccbce83f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d82b81f-d252-4fd6-ac51-bc4dccbce83f-maas-api-openshift-ca\") pod \"1d82b81f-d252-4fd6-ac51-bc4dccbce83f\" (UID: \"1d82b81f-d252-4fd6-ac51-bc4dccbce83f\") "
Oct 30 22:04:52.682430 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:52.682115    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d82b81f-d252-4fd6-ac51-bc4dccbce83f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d82b81f-d252-4fd6-ac51-bc4dccbce83f-maas-api-xks-ca\") pod \"1d82b81f-d252-4fd6-ac51-bc4dccbce83f\" (UID: \"1d82b81f-d252-4fd6-ac51-bc4dccbce83f\") "
Oct 30 22:04:52.682430 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:52.682238    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d82b81f-d252-4fd6-ac51-bc4dccbce83f-kube-api-access-zljt6\" (UniqueName: \"kubernetes.io/projected/1d82b81f-d252-4fd6-ac51-bc4dccbce83f-kube-api-access-zljt6\") pod \"1d82b81f-d252-4fd6-ac51-bc4dccbce83f\" (UID: \"1d82b81f-d252-4fd6-ac51-bc4dccbce83f\") "
Oct 30 22:04:52.682849 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:52.682485    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d82b81f-d252-4fd6-ac51-bc4dccbce83f-maas-api-openshift-ca" pod "1d82b81f-d252-4fd6-ac51-bc4dccbce83f" (UID: "1d82b81f-d252-4fd6-ac51-bc4dccbce83f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:04:52.684287 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:52.683652    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d82b81f-d252-4fd6-ac51-bc4dccbce83f-maas-api-openshift-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:04:52.688962 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:52.688936    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d82b81f-d252-4fd6-ac51-bc4dccbce83f-maas-api-xks-ca" pod "1d82b81f-d252-4fd6-ac51-bc4dccbce83f" (UID: "1d82b81f-d252-4fd6-ac51-bc4dccbce83f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:04:52.689065 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:52.689036    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d82b81f-d252-4fd6-ac51-bc4dccbce83f-kube-api-access-zljt6" pod "1d82b81f-d252-4fd6-ac51-bc4dccbce83f" (UID: "1d82b81f-d252-4fd6-ac51-bc4dccbce83f"). InnerVolumeSpecName "kube-api-access-zljt6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:04:52.784795 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:52.784765    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d82b81f-d252-4fd6-ac51-bc4dccbce83f-maas-api-xks-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:04:52.784795 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:52.784794    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zljt6\" (UniqueName: \"kubernetes.io/projected/1d82b81f-d252-4fd6-ac51-bc4dccbce83f-kube-api-access-zljt6\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:04:52.907441 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:52.907366    5061 patch_prober.go:29] interesting pod/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-4j4xr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body=
Oct 30 22:04:52.907614 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:52.907438    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-4j4xr" podUID="aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused"
Oct 30 22:04:53.158745 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:53.158673    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37cbad61-v9npg"
Oct 30 22:04:53.158745 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:53.158702    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37cbad61-v9npg" event={"ID":"1d82b81f-d252-4fd6-ac51-bc4dccbce83f","Type":"ContainerDied","Data":"e28cfa386fe76f1201f93b6b917b69160040c1d53544546c7b5597a7494d8783"}
Oct 30 22:04:53.158972 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:53.158767    5061 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e28cfa386fe76f1201f93b6b917b69160040c1d53544546c7b5597a7494d8783"
Oct 30 22:04:53.680946 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:53.680897    5061 patch_prober.go:29] interesting pod/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-bpksk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Oct 30 22:04:53.681165 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:53.680963    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-bpksk" podUID="c88c84eb-d49e-49f8-b26e-429659b48ad6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Oct 30 22:04:53.906924 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:53.906642    5061 patch_prober.go:29] interesting pod/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-4j4xr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body=
Oct 30 22:04:53.906924 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:53.906701    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-4j4xr" podUID="aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused"
Oct 30 22:04:54.198279 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:54.198243    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:04:54.198415 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:54.198308    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:04:54.669587 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:04:54.669502    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 22:04:54.684224 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:54.684201    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-bpksk"
Oct 30 22:04:54.684738 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:54.684714    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-bpksk"
Oct 30 22:04:54.685472 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:54.685453    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-bpksk"
Oct 30 22:04:55.097718 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:55.097685    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-4j4xr"
Oct 30 22:04:55.099097 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:55.099073    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-4j4xr"
Oct 30 22:04:55.101010 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:55.100980    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-4j4xr"
Oct 30 22:04:55.179023 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:55.178986    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c586c" event={"ID":"dc212ddd-b71e-44c3-88f2-f7c0936bf7d4","Type":"ContainerStarted","Data":"4ae4b0264f6e207c137533a7efb2e4a0332771b10321bfa1e5bf7a7e9f57aba1"}
Oct 30 22:04:55.179309 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:55.179279    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c586c"
Oct 30 22:04:55.181257 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:55.181232    5061 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-c586c container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:04:55.181344 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:55.181283    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c586c" podUID="dc212ddd-b71e-44c3-88f2-f7c0936bf7d4" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:04:56.187859 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:56.187828    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c586c"
Oct 30 22:04:56.349682 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:04:56.349630    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c586c" podStartSLOduration=4.246353104 podStartE2EDuration="11.349616157s" totalImagesPullingTime="7.103263053s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:55.465832406 +0000 UTC m=+1349.078602164" watchObservedRunningTime="2026-10-30 22:04:56.349616157 +0000 UTC m=+1349.962385915"
Oct 30 22:05:24.887585 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:05:24.886917    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 22:05:25.355551 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:05:25.355266    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a07620/simulator-w1-a07620-kserve-55b44674f4-4z9lk" podUID="efe7c063-fb0e-4679-8a7c-13f2a0abaa74" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:05:25.394551 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:05:25.393548    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-17d809/distinct-w4-17d809-kserve-6b8f5c48cd-lhrwp" podUID="ef5e8691-42e1-471b-ba7d-90881028eca7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:05:25.501551 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:05:25.500699    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2d1f91/simulator-w0-2d1f91-kserve-7699765c74-jwtkb" podUID="44b7d6f6-32d8-4ef4-95bc-eef604fa6b5b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:05:25.501551 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:05:25.501029    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-14af5c/distinct-w6-14af5c-kserve-7c7cb9d69c-hljl9" podUID="1f8d9486-fb2f-491a-a8d7-c23c21bd116a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded"
Oct 30 22:05:25.625613 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:05:25.624673    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2d1f91/premium-w0-2d1f91-kserve-6965b558cf-59lgw" podUID="fb84233d-25e0-48bc-8899-c3a3cb3cfeb9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:05:25.625613 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:05:25.625069    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-14af5c/distinct-2-w6-14af5c-kserve-58d554bdb6-r5r6j" podUID="990f4e94-708e-45f3-b2ad-01cd50b3257b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:05:25.806552 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:05:25.805630    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-17d809/premium-w4-17d809-kserve-7c9bdbbcbc-5mtmm" podUID="1e0eca81-7db1-482b-8dd6-5b28fb89e010" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:05:25.814674 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:05:25.814623    5061 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-5be0d1/premium-w2-5be0d1-kserve-759b6fb995-c6gp4" podUID="933f2470-68de-4768-a05b-6c454f16f5f2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:05:26.012609 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:05:25.985622    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5be0d1/distinct-w2-5be0d1-kserve-6b85698c8-sxn9k" podUID="7a2dbfc5-908f-41d1-baff-074c56d2cf16" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:05:26.196689 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:05:26.195063    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-17d809/simulator-w4-17d809-kserve-78d67b6f4-7hsvb" podUID="e7337cf8-4f59-4afb-9b50-6b6c9fa4a773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:05:26.230552 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:05:26.217980    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5be0d1/premium-w2-5be0d1-kserve-759b6fb995-c6gp4" podUID="933f2470-68de-4768-a05b-6c454f16f5f2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded"
Oct 30 22:05:26.319842 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:05:26.317162    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/distinct-2-w3-f776cb-kserve-7b4d4cc954-d88pf" podUID="5b184062-7fac-43d4-ba5f-21567df85da3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded"
Oct 30 22:05:26.362626 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:05:26.356093    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a07620/simulator-w1-a07620-kserve-55b44674f4-4z9lk" podUID="efe7c063-fb0e-4679-8a7c-13f2a0abaa74" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:05:26.362626 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:05:26.356437    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a07620/simulator-w1-a07620-kserve-55b44674f4-4z9lk"
Oct 30 22:05:26.395964 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:05:26.394871    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-17d809/distinct-w4-17d809-kserve-6b8f5c48cd-lhrwp" podUID="ef5e8691-42e1-471b-ba7d-90881028eca7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:05:26.401546 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:05:26.396253    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-17d809/distinct-w4-17d809-kserve-6b8f5c48cd-lhrwp"
Oct 30 22:05:26.518711 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:05:26.518664    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-14af5c/distinct-w6-14af5c-kserve-7c7cb9d69c-hljl9" podUID="1f8d9486-fb2f-491a-a8d7-c23c21bd116a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:05:26.520946 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:05:26.520910    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-blrrb" podUID="cb4b1ed5-2192-4877-8be7-ca82af831af5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": context deadline exceeded"
Oct 30 22:05:26.617603 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:05:26.613208    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-14af5c/premium-w6-14af5c-kserve-6549647f4-f4gwg" podUID="9b42ee01-c60e-462c-a0e0-b04a52014db6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded"
Oct 30 22:05:26.617603 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:05:26.613753    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2d1f91/simulator-w0-2d1f91-kserve-7699765c74-jwtkb" podUID="44b7d6f6-32d8-4ef4-95bc-eef604fa6b5b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 22:05:26.617603 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:05:26.614048    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-14af5c/distinct-w6-14af5c-kserve-7c7cb9d69c-hljl9"
Oct 30 22:05:26.617603 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:05:26.614169    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2d1f91/simulator-w0-2d1f91-kserve-7699765c74-jwtkb"
Oct 30 22:05:26.658550 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:05:26.653930    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2d1f91/premium-w0-2d1f91-kserve-6965b558cf-59lgw" podUID="fb84233d-25e0-48bc-8899-c3a3cb3cfeb9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 30 22:05:26.658550 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:05:26.654028    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-14af5c/distinct-2-w6-14af5c-kserve-58d554bdb6-r5r6j" podUID="990f4e94-708e-45f3-b2ad-01cd50b3257b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded"
Oct 30 22:05:26.902607 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:05:26.834841    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-17d809/premium-w4-17d809-kserve-7c9bdbbcbc-5mtmm" podUID="1e0eca81-7db1-482b-8dd6-5b28fb89e010" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:05:27.048550 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:05:27.039701    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5be0d1/distinct-w2-5be0d1-kserve-6b85698c8-sxn9k" podUID="7a2dbfc5-908f-41d1-baff-074c56d2cf16" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:05:27.286660 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:05:27.195310    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-14af5c/simulator-w6-14af5c-kserve-85b8bd96d7-wlcb9" podUID="39766c7d-5f1d-42db-8920-c0321cd12c99" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:05:27.286660 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:05:27.268807    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-17d809/simulator-w4-17d809-kserve-78d67b6f4-7hsvb" podUID="e7337cf8-4f59-4afb-9b50-6b6c9fa4a773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Oct 30 22:05:27.286660 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:05:27.269448    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5be0d1/premium-w2-5be0d1-kserve-759b6fb995-c6gp4" podUID="933f2470-68de-4768-a05b-6c454f16f5f2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded"
Oct 30 22:05:27.613514 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:05:27.573097    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a07620/simulator-w1-a07620-kserve-55b44674f4-4z9lk" podUID="efe7c063-fb0e-4679-8a7c-13f2a0abaa74" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Oct 30 22:05:27.613514 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:05:27.598447    5061 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-blrrb" podUID="cb4b1ed5-2192-4877-8be7-ca82af831af5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/health\": context deadline exceeded"
Oct 30 22:05:27.635606 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:05:27.630677    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2d1f91/simulator-w0-2d1f91-kserve-7699765c74-jwtkb" podUID="44b7d6f6-32d8-4ef4-95bc-eef604fa6b5b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:05:27.676546 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:05:27.676084    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-17d809/distinct-w4-17d809-kserve-6b8f5c48cd-lhrwp" podUID="ef5e8691-42e1-471b-ba7d-90881028eca7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded"
Oct 30 22:05:27.676546 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:05:27.676493    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-blrrb" podUID="cb4b1ed5-2192-4877-8be7-ca82af831af5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:05:27.683870 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:05:27.679440    5061 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.065s"
Oct 30 22:05:27.683870 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:05:27.679559    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2d1f91/premium-w0-2d1f91-kserve-6965b558cf-59lgw"
Oct 30 22:05:27.683870 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:05:27.679657    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-14af5c/distinct-2-w6-14af5c-kserve-58d554bdb6-r5r6j"
Oct 30 22:05:27.683870 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:05:27.679808    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-17d809/premium-w4-17d809-kserve-7c9bdbbcbc-5mtmm"
Oct 30 22:05:27.683870 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:05:27.679866    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-5be0d1/distinct-w2-5be0d1-kserve-6b85698c8-sxn9k"
Oct 30 22:05:27.683870 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:05:27.679929    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-17d809/simulator-w4-17d809-kserve-78d67b6f4-7hsvb"
Oct 30 22:05:27.683870 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:05:27.679991    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-5be0d1/premium-w2-5be0d1-kserve-759b6fb995-c6gp4"
Oct 30 22:05:27.683870 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:05:27.680040    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-14af5c/distinct-w6-14af5c-kserve-7c7cb9d69c-hljl9"
Oct 30 22:05:27.683870 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:05:27.680081    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-17d809/premium-w4-17d809-kserve-7c9bdbbcbc-5mtmm"
Oct 30 22:05:27.683870 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:05:27.680123    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2d1f91/premium-w0-2d1f91-kserve-6965b558cf-59lgw"
Oct 30 22:05:27.686860 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:05:27.686836    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-blrrb"
Oct 30 22:05:27.802844 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:05:27.802793    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/distinct-2-w3-f776cb-kserve-7b4d4cc954-d88pf" podUID="5b184062-7fac-43d4-ba5f-21567df85da3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded"
Oct 30 22:05:27.803423 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:05:27.803386    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-14af5c/premium-w6-14af5c-kserve-6549647f4-f4gwg" podUID="9b42ee01-c60e-462c-a0e0-b04a52014db6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:05:27.818297 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:05:27.818272    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f776cb/distinct-2-w3-f776cb-kserve-7b4d4cc954-d88pf"
Oct 30 22:05:27.818563 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:05:27.818544    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-14af5c/premium-w6-14af5c-kserve-6549647f4-f4gwg"
Oct 30 22:05:27.907513 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:05:27.907428    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-14af5c/distinct-2-w6-14af5c-kserve-58d554bdb6-r5r6j"
Oct 30 22:05:27.955146 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:05:27.955114    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-17d809/simulator-w4-17d809-kserve-78d67b6f4-7hsvb"
Oct 30 22:05:27.955321 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:05:27.955177    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2d1f91/simulator-w0-2d1f91-kserve-7699765c74-jwtkb"
Oct 30 22:05:27.955321 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:05:27.955212    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a07620/simulator-w1-a07620-kserve-55b44674f4-4z9lk"
Oct 30 22:05:27.955321 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:05:27.955241    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-14af5c/premium-w6-14af5c-kserve-6549647f4-f4gwg"
Oct 30 22:05:27.955321 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:05:27.955269    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-5be0d1/premium-w2-5be0d1-kserve-759b6fb995-c6gp4"
Oct 30 22:05:27.955321 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:05:27.955298    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-5be0d1/distinct-w2-5be0d1-kserve-6b85698c8-sxn9k"
Oct 30 22:05:27.955321 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:05:27.955326    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f776cb/distinct-2-w3-f776cb-kserve-7b4d4cc954-d88pf"
Oct 30 22:05:27.957821 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:05:27.957801    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-blrrb"
Oct 30 22:05:27.996228 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:05:27.986519    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-17d809/distinct-w4-17d809-kserve-6b8f5c48cd-lhrwp"
Oct 30 22:05:28.910409 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:05:28.910377    5061 generic.go:356] "Generic (PLEG): container finished" podID="661f1c2f-0035-455b-8dfe-21d798d35552" containerID="c3dd2a5a9ef8e07e73b7bd91cd7168349da30625a3806f69b8d79e575826e91a" exitCode=0
Oct 30 22:05:28.910826 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:05:28.910450    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-llrwj-vhl5p" event={"ID":"661f1c2f-0035-455b-8dfe-21d798d35552","Type":"ContainerDied","Data":"c3dd2a5a9ef8e07e73b7bd91cd7168349da30625a3806f69b8d79e575826e91a"}
Oct 30 22:05:30.163312 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:05:30.163279    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-llrwj-vhl5p"
Oct 30 22:05:30.331740 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:05:30.331708    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/661f1c2f-0035-455b-8dfe-21d798d35552-kube-api-access-5785c\" (UniqueName: \"kubernetes.io/projected/661f1c2f-0035-455b-8dfe-21d798d35552-kube-api-access-5785c\") pod \"661f1c2f-0035-455b-8dfe-21d798d35552\" (UID: \"661f1c2f-0035-455b-8dfe-21d798d35552\") "
Oct 30 22:05:30.332020 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:05:30.332000    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/661f1c2f-0035-455b-8dfe-21d798d35552-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/661f1c2f-0035-455b-8dfe-21d798d35552-archives-path\") pod \"661f1c2f-0035-455b-8dfe-21d798d35552\" (UID: \"661f1c2f-0035-455b-8dfe-21d798d35552\") "
Oct 30 22:05:30.332120 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:05:30.332057    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/661f1c2f-0035-455b-8dfe-21d798d35552-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/661f1c2f-0035-455b-8dfe-21d798d35552-service-ca-bundle\") pod \"661f1c2f-0035-455b-8dfe-21d798d35552\" (UID: \"661f1c2f-0035-455b-8dfe-21d798d35552\") "
Oct 30 22:05:30.332120 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:05:30.332092    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/661f1c2f-0035-455b-8dfe-21d798d35552-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/661f1c2f-0035-455b-8dfe-21d798d35552-trusted-ca-bundle\") pod \"661f1c2f-0035-455b-8dfe-21d798d35552\" (UID: \"661f1c2f-0035-455b-8dfe-21d798d35552\") "
Oct 30 22:05:30.332489 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:05:30.332462    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/661f1c2f-0035-455b-8dfe-21d798d35552-archives-path" pod "661f1c2f-0035-455b-8dfe-21d798d35552" (UID: "661f1c2f-0035-455b-8dfe-21d798d35552"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:05:30.332596 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:05:30.332485    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/661f1c2f-0035-455b-8dfe-21d798d35552-service-ca-bundle" pod "661f1c2f-0035-455b-8dfe-21d798d35552" (UID: "661f1c2f-0035-455b-8dfe-21d798d35552"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:05:30.332596 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:05:30.332573    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/661f1c2f-0035-455b-8dfe-21d798d35552-trusted-ca-bundle" pod "661f1c2f-0035-455b-8dfe-21d798d35552" (UID: "661f1c2f-0035-455b-8dfe-21d798d35552"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:05:30.332801 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:05:30.332787    5061 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/661f1c2f-0035-455b-8dfe-21d798d35552-archives-path\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:05:30.332840 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:05:30.332804    5061 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/661f1c2f-0035-455b-8dfe-21d798d35552-service-ca-bundle\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:05:30.332840 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:05:30.332813    5061 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/661f1c2f-0035-455b-8dfe-21d798d35552-trusted-ca-bundle\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:05:30.333984 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:05:30.333961    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/661f1c2f-0035-455b-8dfe-21d798d35552-kube-api-access-5785c" pod "661f1c2f-0035-455b-8dfe-21d798d35552" (UID: "661f1c2f-0035-455b-8dfe-21d798d35552"). InnerVolumeSpecName "kube-api-access-5785c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:05:30.434858 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:05:30.434821    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5785c\" (UniqueName: \"kubernetes.io/projected/661f1c2f-0035-455b-8dfe-21d798d35552-kube-api-access-5785c\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:05:30.930612 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:05:30.930572    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-llrwj-vhl5p" event={"ID":"661f1c2f-0035-455b-8dfe-21d798d35552","Type":"ContainerDied","Data":"c8f0c6ee2e808b0b42f614df38bb609f6e000c2915ced8813ad735e730cc5872"}
Oct 30 22:05:30.930612 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:05:30.930600    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-llrwj-vhl5p"
Oct 30 22:05:30.930838 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:05:30.930635    5061 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c8f0c6ee2e808b0b42f614df38bb609f6e000c2915ced8813ad735e730cc5872"
Oct 30 22:05:54.668236 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:05:54.668156    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 22:06:14.299907 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:14.299869    5061 generic.go:356] "Generic (PLEG): container finished" podID="14ca670b-dd10-4d17-99d0-b1497d64da96" containerID="e265965f3aa420f19497d908d456f7ab0c9b417c759ff5030ffe22eb68186dad" exitCode=0
Oct 30 22:06:14.300453 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:14.299962    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b3084d7f-openshift-default-5b87dddbbf-5h4fs" event={"ID":"14ca670b-dd10-4d17-99d0-b1497d64da96","Type":"ContainerDied","Data":"e265965f3aa420f19497d908d456f7ab0c9b417c759ff5030ffe22eb68186dad"}
Oct 30 22:06:14.300571 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:14.300450    5061 scope.go:122] "RemoveContainer" containerID="e265965f3aa420f19497d908d456f7ab0c9b417c759ff5030ffe22eb68186dad"
Oct 30 22:06:14.302059 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:14.302029    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:06:14.302163 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:14.302095    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:06:14.302163 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:14.302128    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:06:15.334091 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:15.334047    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b3084d7f-openshift-default-5b87dddbbf-5h4fs" event={"ID":"14ca670b-dd10-4d17-99d0-b1497d64da96","Type":"ContainerStarted","Data":"248c7bb3000f571b2427ea712d0efd834e643fc9d96fb5f8d21a6b2f144a8dc7"}
Oct 30 22:06:15.715605 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:15.707796    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-b3084d7f-openshift-default-5b87dddbbf-5h4fs"
Oct 30 22:06:15.730561 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:15.729638    5061 patch_prober.go:29] interesting pod/e2e-ait-b3084d7f-openshift-default-5b87dddbbf-5h4fs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body=
Oct 30 22:06:15.730561 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:15.729719    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b3084d7f-openshift-default-5b87dddbbf-5h4fs" podUID="14ca670b-dd10-4d17-99d0-b1497d64da96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused"
Oct 30 22:06:15.970341 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:15.970238    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-b3084d7f-openshift-default-5b87dddbbf-5h4fs"
Oct 30 22:06:16.707349 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:16.707317    5061 patch_prober.go:29] interesting pod/e2e-ait-b3084d7f-openshift-default-5b87dddbbf-5h4fs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body=
Oct 30 22:06:16.707843 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:16.707371    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b3084d7f-openshift-default-5b87dddbbf-5h4fs" podUID="14ca670b-dd10-4d17-99d0-b1497d64da96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused"
Oct 30 22:06:17.732999 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:17.732957    5061 patch_prober.go:29] interesting pod/e2e-ait-b3084d7f-openshift-default-5b87dddbbf-5h4fs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body=
Oct 30 22:06:17.733398 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:17.733038    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b3084d7f-openshift-default-5b87dddbbf-5h4fs" podUID="14ca670b-dd10-4d17-99d0-b1497d64da96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused"
Oct 30 22:06:18.707236 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:18.707201    5061 patch_prober.go:29] interesting pod/e2e-ait-b3084d7f-openshift-default-5b87dddbbf-5h4fs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body=
Oct 30 22:06:18.707409 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:18.707252    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b3084d7f-openshift-default-5b87dddbbf-5h4fs" podUID="14ca670b-dd10-4d17-99d0-b1497d64da96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused"
Oct 30 22:06:19.707394 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:19.707321    5061 patch_prober.go:29] interesting pod/e2e-ait-b3084d7f-openshift-default-5b87dddbbf-5h4fs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body=
Oct 30 22:06:19.707394 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:19.707380    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b3084d7f-openshift-default-5b87dddbbf-5h4fs" podUID="14ca670b-dd10-4d17-99d0-b1497d64da96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused"
Oct 30 22:06:20.712031 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:20.711999    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-b3084d7f-openshift-default-5b87dddbbf-5h4fs"
Oct 30 22:06:20.713060 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:20.713041    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-b3084d7f-openshift-default-5b87dddbbf-5h4fs"
Oct 30 22:06:24.519292 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:24.519246    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-p6nk7"]
Oct 30 22:06:24.520177 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:24.520148    5061 cpu_manager.go:414] "containerMap: removing container" podUID="1d82b81f-d252-4fd6-ac51-bc4dccbce83f" containerName="revoke-keys"
Oct 30 22:06:24.520177 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:24.520173    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d82b81f-d252-4fd6-ac51-bc4dccbce83f" containerName="revoke-keys"
Oct 30 22:06:24.520364 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:24.520197    5061 cpu_manager.go:414] "containerMap: removing container" podUID="661f1c2f-0035-455b-8dfe-21d798d35552" containerName="insights-gathering"
Oct 30 22:06:24.520364 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:24.520205    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="661f1c2f-0035-455b-8dfe-21d798d35552" containerName="insights-gathering"
Oct 30 22:06:24.520434 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:24.520389    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d82b81f-d252-4fd6-ac51-bc4dccbce83f" containerName="revoke-keys"
Oct 30 22:06:24.520434 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:24.520402    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="661f1c2f-0035-455b-8dfe-21d798d35552" containerName="insights-gathering"
Oct 30 22:06:24.524279 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:24.524254    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-p6nk7"
Oct 30 22:06:24.555843 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:24.555813    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c31e17a7-abf1-4552-8ec5-d9edc8e11314-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-p6nk7\" (UID: \"c31e17a7-abf1-4552-8ec5-d9edc8e11314\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-p6nk7"
Oct 30 22:06:24.555958 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:24.555894    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c31e17a7-abf1-4552-8ec5-d9edc8e11314-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-p6nk7\" (UID: \"c31e17a7-abf1-4552-8ec5-d9edc8e11314\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-p6nk7"
Oct 30 22:06:24.556218 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:24.556191    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c31e17a7-abf1-4552-8ec5-d9edc8e11314-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-p6nk7\" (UID: \"c31e17a7-abf1-4552-8ec5-d9edc8e11314\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-p6nk7"
Oct 30 22:06:24.556218 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:24.556252    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c31e17a7-abf1-4552-8ec5-d9edc8e11314-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-p6nk7\" (UID: \"c31e17a7-abf1-4552-8ec5-d9edc8e11314\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-p6nk7"
Oct 30 22:06:24.556454 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:24.556279    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdmss\" (UniqueName: \"kubernetes.io/projected/c31e17a7-abf1-4552-8ec5-d9edc8e11314-kube-api-access-qdmss\") pod \"maas-default-gateway-openshift-default-694db55c87-p6nk7\" (UID: \"c31e17a7-abf1-4552-8ec5-d9edc8e11314\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-p6nk7"
Oct 30 22:06:24.556454 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:24.556414    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c31e17a7-abf1-4552-8ec5-d9edc8e11314-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-p6nk7\" (UID: \"c31e17a7-abf1-4552-8ec5-d9edc8e11314\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-p6nk7"
Oct 30 22:06:24.556569 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:24.556472    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c31e17a7-abf1-4552-8ec5-d9edc8e11314-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-p6nk7\" (UID: \"c31e17a7-abf1-4552-8ec5-d9edc8e11314\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-p6nk7"
Oct 30 22:06:24.556569 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:24.556505    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c31e17a7-abf1-4552-8ec5-d9edc8e11314-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-p6nk7\" (UID: \"c31e17a7-abf1-4552-8ec5-d9edc8e11314\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-p6nk7"
Oct 30 22:06:24.556664 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:24.556590    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c31e17a7-abf1-4552-8ec5-d9edc8e11314-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-p6nk7\" (UID: \"c31e17a7-abf1-4552-8ec5-d9edc8e11314\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-p6nk7"
Oct 30 22:06:24.575468 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:24.575444    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-d6th4\""
Oct 30 22:06:24.628266 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:24.628238    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-p6nk7"]
Oct 30 22:06:24.658373 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:24.658337    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c31e17a7-abf1-4552-8ec5-d9edc8e11314-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-p6nk7\" (UID: \"c31e17a7-abf1-4552-8ec5-d9edc8e11314\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-p6nk7"
Oct 30 22:06:24.658518 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:24.658386    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdmss\" (UniqueName: \"kubernetes.io/projected/c31e17a7-abf1-4552-8ec5-d9edc8e11314-kube-api-access-qdmss\") pod \"maas-default-gateway-openshift-default-694db55c87-p6nk7\" (UID: \"c31e17a7-abf1-4552-8ec5-d9edc8e11314\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-p6nk7"
Oct 30 22:06:24.658518 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:24.658483    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c31e17a7-abf1-4552-8ec5-d9edc8e11314-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-p6nk7\" (UID: \"c31e17a7-abf1-4552-8ec5-d9edc8e11314\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-p6nk7"
Oct 30 22:06:24.658624 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:24.658569    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c31e17a7-abf1-4552-8ec5-d9edc8e11314-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-p6nk7\" (UID: \"c31e17a7-abf1-4552-8ec5-d9edc8e11314\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-p6nk7"
Oct 30 22:06:24.658624 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:24.658611    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c31e17a7-abf1-4552-8ec5-d9edc8e11314-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-p6nk7\" (UID: \"c31e17a7-abf1-4552-8ec5-d9edc8e11314\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-p6nk7"
Oct 30 22:06:24.658700 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:24.658673    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c31e17a7-abf1-4552-8ec5-d9edc8e11314-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-p6nk7\" (UID: \"c31e17a7-abf1-4552-8ec5-d9edc8e11314\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-p6nk7"
Oct 30 22:06:24.658835 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:24.658814    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c31e17a7-abf1-4552-8ec5-d9edc8e11314-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-p6nk7\" (UID: \"c31e17a7-abf1-4552-8ec5-d9edc8e11314\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-p6nk7"
Oct 30 22:06:24.658912 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:24.658885    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c31e17a7-abf1-4552-8ec5-d9edc8e11314-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-p6nk7\" (UID: \"c31e17a7-abf1-4552-8ec5-d9edc8e11314\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-p6nk7"
Oct 30 22:06:24.658972 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:24.658921    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c31e17a7-abf1-4552-8ec5-d9edc8e11314-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-p6nk7\" (UID: \"c31e17a7-abf1-4552-8ec5-d9edc8e11314\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-p6nk7"
Oct 30 22:06:24.659077 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:24.659057    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c31e17a7-abf1-4552-8ec5-d9edc8e11314-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-p6nk7\" (UID: \"c31e17a7-abf1-4552-8ec5-d9edc8e11314\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-p6nk7"
Oct 30 22:06:24.659203 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:24.659163    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c31e17a7-abf1-4552-8ec5-d9edc8e11314-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-p6nk7\" (UID: \"c31e17a7-abf1-4552-8ec5-d9edc8e11314\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-p6nk7"
Oct 30 22:06:24.659260 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:24.659233    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c31e17a7-abf1-4552-8ec5-d9edc8e11314-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-p6nk7\" (UID: \"c31e17a7-abf1-4552-8ec5-d9edc8e11314\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-p6nk7"
Oct 30 22:06:24.659304 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:24.659272    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c31e17a7-abf1-4552-8ec5-d9edc8e11314-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-p6nk7\" (UID: \"c31e17a7-abf1-4552-8ec5-d9edc8e11314\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-p6nk7"
Oct 30 22:06:24.659568 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:24.659507    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c31e17a7-abf1-4552-8ec5-d9edc8e11314-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-p6nk7\" (UID: \"c31e17a7-abf1-4552-8ec5-d9edc8e11314\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-p6nk7"
Oct 30 22:06:24.661141 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:24.661121    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c31e17a7-abf1-4552-8ec5-d9edc8e11314-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-p6nk7\" (UID: \"c31e17a7-abf1-4552-8ec5-d9edc8e11314\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-p6nk7"
Oct 30 22:06:24.662079 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:24.662057    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c31e17a7-abf1-4552-8ec5-d9edc8e11314-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-p6nk7\" (UID: \"c31e17a7-abf1-4552-8ec5-d9edc8e11314\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-p6nk7"
Oct 30 22:06:24.676753 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:06:24.676721    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 22:06:24.715222 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:24.714585    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c31e17a7-abf1-4552-8ec5-d9edc8e11314-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-p6nk7\" (UID: \"c31e17a7-abf1-4552-8ec5-d9edc8e11314\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-p6nk7"
Oct 30 22:06:24.715222 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:24.715172    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdmss\" (UniqueName: \"kubernetes.io/projected/c31e17a7-abf1-4552-8ec5-d9edc8e11314-kube-api-access-qdmss\") pod \"maas-default-gateway-openshift-default-694db55c87-p6nk7\" (UID: \"c31e17a7-abf1-4552-8ec5-d9edc8e11314\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-p6nk7"
Oct 30 22:06:24.840385 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:24.840355    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-p6nk7"
Oct 30 22:06:25.265391 ip-10-0-130-60 kubenswrapper[5061]: W1030 22:06:25.265361    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc31e17a7_abf1_4552_8ec5_d9edc8e11314.slice/crio-7ddad95bfafe11329fcb1dfce9548748c5a565bf1326559a026871f5c399a4fc WatchSource:0}: Error finding container 7ddad95bfafe11329fcb1dfce9548748c5a565bf1326559a026871f5c399a4fc: Status 404 returned error can't find the container with id 7ddad95bfafe11329fcb1dfce9548748c5a565bf1326559a026871f5c399a4fc
Oct 30 22:06:25.266777 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:25.266751    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-p6nk7"]
Oct 30 22:06:25.267963 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:25.267927    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:06:25.268057 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:25.268026    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:06:25.268126 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:25.268074    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:06:25.415212 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:25.415178    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-p6nk7" event={"ID":"c31e17a7-abf1-4552-8ec5-d9edc8e11314","Type":"ContainerStarted","Data":"ebe5e3c0b2ca1d3b8517024f2ad6ea626af9efcde6ec6e2d730c09ca0e613434"}
Oct 30 22:06:25.415402 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:25.415228    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-p6nk7" event={"ID":"c31e17a7-abf1-4552-8ec5-d9edc8e11314","Type":"ContainerStarted","Data":"7ddad95bfafe11329fcb1dfce9548748c5a565bf1326559a026871f5c399a4fc"}
Oct 30 22:06:25.530291 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:25.530194    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-p6nk7" podStartSLOduration=1.530180536 podStartE2EDuration="1.530180536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:06:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:25.527191894 +0000 UTC m=+1439.139961651" watchObservedRunningTime="2026-10-30 22:06:25.530180536 +0000 UTC m=+1439.142950294"
Oct 30 22:06:25.843830 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:25.843773    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-p6nk7"
Oct 30 22:06:26.842991 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:26.842958    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-p6nk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Oct 30 22:06:26.843416 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:26.843026    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-p6nk7" podUID="c31e17a7-abf1-4552-8ec5-d9edc8e11314" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Oct 30 22:06:27.841576 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:27.841518    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-p6nk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Oct 30 22:06:27.841773 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:27.841595    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-p6nk7" podUID="c31e17a7-abf1-4552-8ec5-d9edc8e11314" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Oct 30 22:06:28.840995 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:28.840957    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-p6nk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Oct 30 22:06:28.841483 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:28.841016    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-p6nk7" podUID="c31e17a7-abf1-4552-8ec5-d9edc8e11314" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Oct 30 22:06:29.841016 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:29.840977    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-p6nk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Oct 30 22:06:29.841431 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:29.841047    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-p6nk7" podUID="c31e17a7-abf1-4552-8ec5-d9edc8e11314" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Oct 30 22:06:30.840784 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:30.840751    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-p6nk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Oct 30 22:06:30.840982 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:30.840808    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-p6nk7" podUID="c31e17a7-abf1-4552-8ec5-d9edc8e11314" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Oct 30 22:06:31.841166 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:31.841125    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-p6nk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Oct 30 22:06:31.841606 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:31.841192    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-p6nk7" podUID="c31e17a7-abf1-4552-8ec5-d9edc8e11314" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Oct 30 22:06:32.841488 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:32.841453    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-p6nk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Oct 30 22:06:32.841940 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:32.841511    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-p6nk7" podUID="c31e17a7-abf1-4552-8ec5-d9edc8e11314" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Oct 30 22:06:33.846766 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:33.846731    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-p6nk7"
Oct 30 22:06:33.847174 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:33.847147    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-p6nk7"
Oct 30 22:06:33.847788 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:33.847770    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-p6nk7"
Oct 30 22:06:39.636523 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:39.636482    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-h8wx4"]
Oct 30 22:06:39.643517 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:39.643488    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-h8wx4"
Oct 30 22:06:39.768550 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:39.766683    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t42hk\" (UniqueName: \"kubernetes.io/projected/8467f23a-1b0c-4d55-a1de-efa04241cf30-kube-api-access-t42hk\") pod \"maas-default-gateway-openshift-default-694db55c87-h8wx4\" (UID: \"8467f23a-1b0c-4d55-a1de-efa04241cf30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-h8wx4"
Oct 30 22:06:39.768550 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:39.766794    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8467f23a-1b0c-4d55-a1de-efa04241cf30-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-h8wx4\" (UID: \"8467f23a-1b0c-4d55-a1de-efa04241cf30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-h8wx4"
Oct 30 22:06:39.768550 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:39.766852    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8467f23a-1b0c-4d55-a1de-efa04241cf30-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-h8wx4\" (UID: \"8467f23a-1b0c-4d55-a1de-efa04241cf30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-h8wx4"
Oct 30 22:06:39.768550 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:39.766906    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8467f23a-1b0c-4d55-a1de-efa04241cf30-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-h8wx4\" (UID: \"8467f23a-1b0c-4d55-a1de-efa04241cf30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-h8wx4"
Oct 30 22:06:39.768550 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:39.767018    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8467f23a-1b0c-4d55-a1de-efa04241cf30-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-h8wx4\" (UID: \"8467f23a-1b0c-4d55-a1de-efa04241cf30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-h8wx4"
Oct 30 22:06:39.768550 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:39.767184    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8467f23a-1b0c-4d55-a1de-efa04241cf30-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-h8wx4\" (UID: \"8467f23a-1b0c-4d55-a1de-efa04241cf30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-h8wx4"
Oct 30 22:06:39.768550 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:39.767349    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8467f23a-1b0c-4d55-a1de-efa04241cf30-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-h8wx4\" (UID: \"8467f23a-1b0c-4d55-a1de-efa04241cf30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-h8wx4"
Oct 30 22:06:39.768550 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:39.767410    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8467f23a-1b0c-4d55-a1de-efa04241cf30-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-h8wx4\" (UID: \"8467f23a-1b0c-4d55-a1de-efa04241cf30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-h8wx4"
Oct 30 22:06:39.768550 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:39.767483    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8467f23a-1b0c-4d55-a1de-efa04241cf30-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-h8wx4\" (UID: \"8467f23a-1b0c-4d55-a1de-efa04241cf30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-h8wx4"
Oct 30 22:06:39.782550 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:39.776788    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7d2bf"]
Oct 30 22:06:39.789843 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:39.789806    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7d2bf"
Oct 30 22:06:39.801130 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:39.801055    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-h8wx4"]
Oct 30 22:06:39.864799 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:39.864751    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7d2bf"]
Oct 30 22:06:39.870985 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:39.870944    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t42hk\" (UniqueName: \"kubernetes.io/projected/8467f23a-1b0c-4d55-a1de-efa04241cf30-kube-api-access-t42hk\") pod \"maas-default-gateway-openshift-default-694db55c87-h8wx4\" (UID: \"8467f23a-1b0c-4d55-a1de-efa04241cf30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-h8wx4"
Oct 30 22:06:39.871597 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:39.871569    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e879c14b-842a-49da-a454-a85d9429b765-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-7d2bf\" (UID: \"e879c14b-842a-49da-a454-a85d9429b765\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7d2bf"
Oct 30 22:06:39.871906 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:39.871888    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e879c14b-842a-49da-a454-a85d9429b765-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-7d2bf\" (UID: \"e879c14b-842a-49da-a454-a85d9429b765\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7d2bf"
Oct 30 22:06:39.872083 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:39.872068    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8467f23a-1b0c-4d55-a1de-efa04241cf30-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-h8wx4\" (UID: \"8467f23a-1b0c-4d55-a1de-efa04241cf30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-h8wx4"
Oct 30 22:06:39.872663 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:39.872640    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e879c14b-842a-49da-a454-a85d9429b765-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-7d2bf\" (UID: \"e879c14b-842a-49da-a454-a85d9429b765\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7d2bf"
Oct 30 22:06:39.872881 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:39.872864    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8467f23a-1b0c-4d55-a1de-efa04241cf30-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-h8wx4\" (UID: \"8467f23a-1b0c-4d55-a1de-efa04241cf30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-h8wx4"
Oct 30 22:06:39.873264 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:39.873241    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e879c14b-842a-49da-a454-a85d9429b765-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-7d2bf\" (UID: \"e879c14b-842a-49da-a454-a85d9429b765\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7d2bf"
Oct 30 22:06:39.873446 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:39.873428    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8467f23a-1b0c-4d55-a1de-efa04241cf30-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-h8wx4\" (UID: \"8467f23a-1b0c-4d55-a1de-efa04241cf30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-h8wx4"
Oct 30 22:06:39.874494 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:39.874474    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8467f23a-1b0c-4d55-a1de-efa04241cf30-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-h8wx4\" (UID: \"8467f23a-1b0c-4d55-a1de-efa04241cf30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-h8wx4"
Oct 30 22:06:39.874952 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:39.874935    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e879c14b-842a-49da-a454-a85d9429b765-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-7d2bf\" (UID: \"e879c14b-842a-49da-a454-a85d9429b765\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7d2bf"
Oct 30 22:06:39.875153 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:39.875134    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pprtc\" (UniqueName: \"kubernetes.io/projected/e879c14b-842a-49da-a454-a85d9429b765-kube-api-access-pprtc\") pod \"maas-default-gateway-openshift-default-694db55c87-7d2bf\" (UID: \"e879c14b-842a-49da-a454-a85d9429b765\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7d2bf"
Oct 30 22:06:39.875351 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:39.875332    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e879c14b-842a-49da-a454-a85d9429b765-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-7d2bf\" (UID: \"e879c14b-842a-49da-a454-a85d9429b765\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7d2bf"
Oct 30 22:06:39.875623 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:39.875601    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8467f23a-1b0c-4d55-a1de-efa04241cf30-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-h8wx4\" (UID: \"8467f23a-1b0c-4d55-a1de-efa04241cf30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-h8wx4"
Oct 30 22:06:39.876396 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:39.876372    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8467f23a-1b0c-4d55-a1de-efa04241cf30-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-h8wx4\" (UID: \"8467f23a-1b0c-4d55-a1de-efa04241cf30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-h8wx4"
Oct 30 22:06:39.878226 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:39.878185    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8467f23a-1b0c-4d55-a1de-efa04241cf30-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-h8wx4\" (UID: \"8467f23a-1b0c-4d55-a1de-efa04241cf30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-h8wx4"
Oct 30 22:06:39.878757 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:39.878729    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e879c14b-842a-49da-a454-a85d9429b765-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-7d2bf\" (UID: \"e879c14b-842a-49da-a454-a85d9429b765\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7d2bf"
Oct 30 22:06:39.879046 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:39.878110    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8467f23a-1b0c-4d55-a1de-efa04241cf30-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-h8wx4\" (UID: \"8467f23a-1b0c-4d55-a1de-efa04241cf30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-h8wx4"
Oct 30 22:06:39.879046 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:39.872568    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8467f23a-1b0c-4d55-a1de-efa04241cf30-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-h8wx4\" (UID: \"8467f23a-1b0c-4d55-a1de-efa04241cf30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-h8wx4"
Oct 30 22:06:39.879046 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:39.874887    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8467f23a-1b0c-4d55-a1de-efa04241cf30-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-h8wx4\" (UID: \"8467f23a-1b0c-4d55-a1de-efa04241cf30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-h8wx4"
Oct 30 22:06:39.879046 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:39.874410    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8467f23a-1b0c-4d55-a1de-efa04241cf30-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-h8wx4\" (UID: \"8467f23a-1b0c-4d55-a1de-efa04241cf30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-h8wx4"
Oct 30 22:06:39.881266 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:39.881241    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8467f23a-1b0c-4d55-a1de-efa04241cf30-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-h8wx4\" (UID: \"8467f23a-1b0c-4d55-a1de-efa04241cf30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-h8wx4"
Oct 30 22:06:39.881410 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:39.878960    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8467f23a-1b0c-4d55-a1de-efa04241cf30-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-h8wx4\" (UID: \"8467f23a-1b0c-4d55-a1de-efa04241cf30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-h8wx4"
Oct 30 22:06:39.881598 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:39.881580    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e879c14b-842a-49da-a454-a85d9429b765-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-7d2bf\" (UID: \"e879c14b-842a-49da-a454-a85d9429b765\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7d2bf"
Oct 30 22:06:39.886363 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:39.886342    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8467f23a-1b0c-4d55-a1de-efa04241cf30-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-h8wx4\" (UID: \"8467f23a-1b0c-4d55-a1de-efa04241cf30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-h8wx4"
Oct 30 22:06:39.888599 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:39.888500    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8467f23a-1b0c-4d55-a1de-efa04241cf30-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-h8wx4\" (UID: \"8467f23a-1b0c-4d55-a1de-efa04241cf30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-h8wx4"
Oct 30 22:06:39.941482 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:39.941444    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8467f23a-1b0c-4d55-a1de-efa04241cf30-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-h8wx4\" (UID: \"8467f23a-1b0c-4d55-a1de-efa04241cf30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-h8wx4"
Oct 30 22:06:39.941679 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:39.941631    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t42hk\" (UniqueName: \"kubernetes.io/projected/8467f23a-1b0c-4d55-a1de-efa04241cf30-kube-api-access-t42hk\") pod \"maas-default-gateway-openshift-default-694db55c87-h8wx4\" (UID: \"8467f23a-1b0c-4d55-a1de-efa04241cf30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-h8wx4"
Oct 30 22:06:39.985933 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:39.984567    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e879c14b-842a-49da-a454-a85d9429b765-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-7d2bf\" (UID: \"e879c14b-842a-49da-a454-a85d9429b765\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7d2bf"
Oct 30 22:06:39.985933 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:39.984628    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e879c14b-842a-49da-a454-a85d9429b765-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-7d2bf\" (UID: \"e879c14b-842a-49da-a454-a85d9429b765\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7d2bf"
Oct 30 22:06:39.985933 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:39.984719    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e879c14b-842a-49da-a454-a85d9429b765-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-7d2bf\" (UID: \"e879c14b-842a-49da-a454-a85d9429b765\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7d2bf"
Oct 30 22:06:39.985933 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:39.984752    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e879c14b-842a-49da-a454-a85d9429b765-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-7d2bf\" (UID: \"e879c14b-842a-49da-a454-a85d9429b765\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7d2bf"
Oct 30 22:06:39.985933 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:39.984799    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e879c14b-842a-49da-a454-a85d9429b765-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-7d2bf\" (UID: \"e879c14b-842a-49da-a454-a85d9429b765\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7d2bf"
Oct 30 22:06:39.985933 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:39.984837    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e879c14b-842a-49da-a454-a85d9429b765-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-7d2bf\" (UID: \"e879c14b-842a-49da-a454-a85d9429b765\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7d2bf"
Oct 30 22:06:39.985933 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:39.984884    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e879c14b-842a-49da-a454-a85d9429b765-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-7d2bf\" (UID: \"e879c14b-842a-49da-a454-a85d9429b765\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7d2bf"
Oct 30 22:06:39.985933 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:39.984931    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pprtc\" (UniqueName: \"kubernetes.io/projected/e879c14b-842a-49da-a454-a85d9429b765-kube-api-access-pprtc\") pod \"maas-default-gateway-openshift-default-694db55c87-7d2bf\" (UID: \"e879c14b-842a-49da-a454-a85d9429b765\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7d2bf"
Oct 30 22:06:39.985933 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:39.984986    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e879c14b-842a-49da-a454-a85d9429b765-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-7d2bf\" (UID: \"e879c14b-842a-49da-a454-a85d9429b765\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7d2bf"
Oct 30 22:06:39.985933 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:39.985564    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e879c14b-842a-49da-a454-a85d9429b765-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-7d2bf\" (UID: \"e879c14b-842a-49da-a454-a85d9429b765\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7d2bf"
Oct 30 22:06:39.985933 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:39.985581    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e879c14b-842a-49da-a454-a85d9429b765-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-7d2bf\" (UID: \"e879c14b-842a-49da-a454-a85d9429b765\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7d2bf"
Oct 30 22:06:39.985933 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:39.985897    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e879c14b-842a-49da-a454-a85d9429b765-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-7d2bf\" (UID: \"e879c14b-842a-49da-a454-a85d9429b765\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7d2bf"
Oct 30 22:06:39.986678 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:39.986449    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e879c14b-842a-49da-a454-a85d9429b765-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-7d2bf\" (UID: \"e879c14b-842a-49da-a454-a85d9429b765\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7d2bf"
Oct 30 22:06:39.986898 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:39.986869    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e879c14b-842a-49da-a454-a85d9429b765-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-7d2bf\" (UID: \"e879c14b-842a-49da-a454-a85d9429b765\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7d2bf"
Oct 30 22:06:39.989228 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:39.989189    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e879c14b-842a-49da-a454-a85d9429b765-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-7d2bf\" (UID: \"e879c14b-842a-49da-a454-a85d9429b765\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7d2bf"
Oct 30 22:06:39.992563 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:39.991176    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e879c14b-842a-49da-a454-a85d9429b765-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-7d2bf\" (UID: \"e879c14b-842a-49da-a454-a85d9429b765\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7d2bf"
Oct 30 22:06:40.029995 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:40.029958    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-h8wx4"
Oct 30 22:06:40.049761 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:40.049429    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e879c14b-842a-49da-a454-a85d9429b765-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-7d2bf\" (UID: \"e879c14b-842a-49da-a454-a85d9429b765\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7d2bf"
Oct 30 22:06:40.051031 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:40.051004    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pprtc\" (UniqueName: \"kubernetes.io/projected/e879c14b-842a-49da-a454-a85d9429b765-kube-api-access-pprtc\") pod \"maas-default-gateway-openshift-default-694db55c87-7d2bf\" (UID: \"e879c14b-842a-49da-a454-a85d9429b765\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7d2bf"
Oct 30 22:06:40.152023 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:40.150816    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7d2bf"
Oct 30 22:06:40.465490 ip-10-0-130-60 kubenswrapper[5061]: W1030 22:06:40.465460    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8467f23a_1b0c_4d55_a1de_efa04241cf30.slice/crio-74df9466ac92c5c622df6c3b8a8341bbfa2b59c03d33f45e01644e6627039a31 WatchSource:0}: Error finding container 74df9466ac92c5c622df6c3b8a8341bbfa2b59c03d33f45e01644e6627039a31: Status 404 returned error can't find the container with id 74df9466ac92c5c622df6c3b8a8341bbfa2b59c03d33f45e01644e6627039a31
Oct 30 22:06:40.465683 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:40.465559    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-h8wx4"]
Oct 30 22:06:40.467920 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:40.467886    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:06:40.468017 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:40.467950    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:06:40.468017 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:40.467979    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:06:40.565006 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:40.564969    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-h8wx4" event={"ID":"8467f23a-1b0c-4d55-a1de-efa04241cf30","Type":"ContainerStarted","Data":"c986b06a87de45bb4e8aab38499e1f8b678031fe7bb60b3c403e00ca54bcaff0"}
Oct 30 22:06:40.565136 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:40.565043    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-h8wx4" event={"ID":"8467f23a-1b0c-4d55-a1de-efa04241cf30","Type":"ContainerStarted","Data":"74df9466ac92c5c622df6c3b8a8341bbfa2b59c03d33f45e01644e6627039a31"}
Oct 30 22:06:40.582610 ip-10-0-130-60 kubenswrapper[5061]: W1030 22:06:40.582583    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode879c14b_842a_49da_a454_a85d9429b765.slice/crio-b125ba27bf3f65a21b2fc6e999a9576cf2b931e9a6c2964c3f2d3e37d5875378 WatchSource:0}: Error finding container b125ba27bf3f65a21b2fc6e999a9576cf2b931e9a6c2964c3f2d3e37d5875378: Status 404 returned error can't find the container with id b125ba27bf3f65a21b2fc6e999a9576cf2b931e9a6c2964c3f2d3e37d5875378
Oct 30 22:06:40.583819 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:40.583790    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7d2bf"]
Oct 30 22:06:40.590709 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:40.590677    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:06:40.590830 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:40.590743    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:06:40.590830 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:40.590772    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:06:40.682592 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:40.682500    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-h8wx4" podStartSLOduration=1.6824815659999999 podStartE2EDuration="1.682481566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:06:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:40.676357145 +0000 UTC m=+1454.289126904" watchObservedRunningTime="2026-10-30 22:06:40.682481566 +0000 UTC m=+1454.295251326"
Oct 30 22:06:41.030627 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:41.030595    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-h8wx4"
Oct 30 22:06:41.031939 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:41.031913    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-h8wx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Oct 30 22:06:41.032071 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:41.031967    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-h8wx4" podUID="8467f23a-1b0c-4d55-a1de-efa04241cf30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Oct 30 22:06:41.578287 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:41.578250    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7d2bf" event={"ID":"e879c14b-842a-49da-a454-a85d9429b765","Type":"ContainerStarted","Data":"6bf51cd427dc7585decf1c3847bba334e9c07d177a22d0d5eac2470d3c75a7d6"}
Oct 30 22:06:41.578492 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:41.578304    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7d2bf" event={"ID":"e879c14b-842a-49da-a454-a85d9429b765","Type":"ContainerStarted","Data":"b125ba27bf3f65a21b2fc6e999a9576cf2b931e9a6c2964c3f2d3e37d5875378"}
Oct 30 22:06:41.705924 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:41.705872    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7d2bf" podStartSLOduration=2.705860173 podStartE2EDuration="2.705860173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:06:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:41.703583694 +0000 UTC m=+1455.316353472" watchObservedRunningTime="2026-10-30 22:06:41.705860173 +0000 UTC m=+1455.318629979"
Oct 30 22:06:42.038730 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:42.038637    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-h8wx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Oct 30 22:06:42.039169 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:42.039139    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-h8wx4" podUID="8467f23a-1b0c-4d55-a1de-efa04241cf30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Oct 30 22:06:42.152550 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:42.152401    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7d2bf"
Oct 30 22:06:42.159721 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:42.159637    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-7d2bf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body=
Oct 30 22:06:42.159721 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:42.159697    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7d2bf" podUID="e879c14b-842a-49da-a454-a85d9429b765" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused"
Oct 30 22:06:43.046552 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:43.041296    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-h8wx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Oct 30 22:06:43.046552 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:43.041368    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-h8wx4" podUID="8467f23a-1b0c-4d55-a1de-efa04241cf30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Oct 30 22:06:43.161549 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:43.161104    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-7d2bf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body=
Oct 30 22:06:43.161549 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:43.161177    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7d2bf" podUID="e879c14b-842a-49da-a454-a85d9429b765" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused"
Oct 30 22:06:44.035551 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:44.034881    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-h8wx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Oct 30 22:06:44.035551 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:44.034940    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-h8wx4" podUID="8467f23a-1b0c-4d55-a1de-efa04241cf30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Oct 30 22:06:44.154557 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:44.153886    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-7d2bf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body=
Oct 30 22:06:44.154557 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:44.153945    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7d2bf" podUID="e879c14b-842a-49da-a454-a85d9429b765" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused"
Oct 30 22:06:45.032551 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:45.030499    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-h8wx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Oct 30 22:06:45.032551 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:45.030572    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-h8wx4" podUID="8467f23a-1b0c-4d55-a1de-efa04241cf30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Oct 30 22:06:45.152449 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:45.152413    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-7d2bf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body=
Oct 30 22:06:45.152857 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:45.152822    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7d2bf" podUID="e879c14b-842a-49da-a454-a85d9429b765" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused"
Oct 30 22:06:46.062022 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:46.061978    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-h8wx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Oct 30 22:06:46.062563 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:46.062045    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-h8wx4" podUID="8467f23a-1b0c-4d55-a1de-efa04241cf30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Oct 30 22:06:46.162751 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:46.162706    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-7d2bf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body=
Oct 30 22:06:46.162960 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:46.162770    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7d2bf" podUID="e879c14b-842a-49da-a454-a85d9429b765" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused"
Oct 30 22:06:47.030225 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:47.030192    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-h8wx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Oct 30 22:06:47.030414 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:47.030256    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-h8wx4" podUID="8467f23a-1b0c-4d55-a1de-efa04241cf30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Oct 30 22:06:47.151519 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:47.151483    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-7d2bf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body=
Oct 30 22:06:47.152020 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:47.151578    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7d2bf" podUID="e879c14b-842a-49da-a454-a85d9429b765" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused"
Oct 30 22:06:48.030842 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:48.030805    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-h8wx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Oct 30 22:06:48.031017 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:48.030861    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-h8wx4" podUID="8467f23a-1b0c-4d55-a1de-efa04241cf30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Oct 30 22:06:48.151795 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:48.151757    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-7d2bf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body=
Oct 30 22:06:48.152170 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:48.151838    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7d2bf" podUID="e879c14b-842a-49da-a454-a85d9429b765" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused"
Oct 30 22:06:49.298899 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:49.298860    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-h8wx4"
Oct 30 22:06:49.300377 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:49.300352    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-h8wx4"
Oct 30 22:06:49.301645 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:49.301621    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-h8wx4"
Oct 30 22:06:49.522080 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:49.522047    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7d2bf"
Oct 30 22:06:49.523487 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:49.523466    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7d2bf"
Oct 30 22:06:49.524363 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:49.524343    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-7d2bf"
Oct 30 22:06:49.832890 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:49.832859    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-4z59k"]
Oct 30 22:06:49.837802 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:49.837780    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-4z59k"
Oct 30 22:06:49.887186 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:49.887160    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-50ff7514-openshift-default-dockercfg-g859w\""
Oct 30 22:06:49.897445 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:49.897409    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-4z59k"]
Oct 30 22:06:49.942442 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:49.942403    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33935040-0f5f-43db-91c7-cae4d0e37369-istio-podinfo\") pod \"e2e-ait-50ff7514-openshift-default-679cf6446f-4z59k\" (UID: \"33935040-0f5f-43db-91c7-cae4d0e37369\") " pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-4z59k"
Oct 30 22:06:49.942442 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:49.942445    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33935040-0f5f-43db-91c7-cae4d0e37369-istio-data\") pod \"e2e-ait-50ff7514-openshift-default-679cf6446f-4z59k\" (UID: \"33935040-0f5f-43db-91c7-cae4d0e37369\") " pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-4z59k"
Oct 30 22:06:49.942752 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:49.942500    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33935040-0f5f-43db-91c7-cae4d0e37369-workload-certs\") pod \"e2e-ait-50ff7514-openshift-default-679cf6446f-4z59k\" (UID: \"33935040-0f5f-43db-91c7-cae4d0e37369\") " pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-4z59k"
Oct 30 22:06:49.942752 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:49.942680    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33935040-0f5f-43db-91c7-cae4d0e37369-workload-socket\") pod \"e2e-ait-50ff7514-openshift-default-679cf6446f-4z59k\" (UID: \"33935040-0f5f-43db-91c7-cae4d0e37369\") " pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-4z59k"
Oct 30 22:06:49.942854 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:49.942798    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33935040-0f5f-43db-91c7-cae4d0e37369-istiod-ca-cert\") pod \"e2e-ait-50ff7514-openshift-default-679cf6446f-4z59k\" (UID: \"33935040-0f5f-43db-91c7-cae4d0e37369\") " pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-4z59k"
Oct 30 22:06:49.942930 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:49.942911    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33935040-0f5f-43db-91c7-cae4d0e37369-istio-token\") pod \"e2e-ait-50ff7514-openshift-default-679cf6446f-4z59k\" (UID: \"33935040-0f5f-43db-91c7-cae4d0e37369\") " pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-4z59k"
Oct 30 22:06:49.942978 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:49.942942    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33935040-0f5f-43db-91c7-cae4d0e37369-credential-socket\") pod \"e2e-ait-50ff7514-openshift-default-679cf6446f-4z59k\" (UID: \"33935040-0f5f-43db-91c7-cae4d0e37369\") " pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-4z59k"
Oct 30 22:06:49.942978 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:49.942975    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4rks\" (UniqueName: \"kubernetes.io/projected/33935040-0f5f-43db-91c7-cae4d0e37369-kube-api-access-t4rks\") pod \"e2e-ait-50ff7514-openshift-default-679cf6446f-4z59k\" (UID: \"33935040-0f5f-43db-91c7-cae4d0e37369\") " pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-4z59k"
Oct 30 22:06:49.943108 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:49.943091    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33935040-0f5f-43db-91c7-cae4d0e37369-istio-envoy\") pod \"e2e-ait-50ff7514-openshift-default-679cf6446f-4z59k\" (UID: \"33935040-0f5f-43db-91c7-cae4d0e37369\") " pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-4z59k"
Oct 30 22:06:50.044870 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:50.044826    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33935040-0f5f-43db-91c7-cae4d0e37369-istio-token\") pod \"e2e-ait-50ff7514-openshift-default-679cf6446f-4z59k\" (UID: \"33935040-0f5f-43db-91c7-cae4d0e37369\") " pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-4z59k"
Oct 30 22:06:50.045073 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:50.044875    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33935040-0f5f-43db-91c7-cae4d0e37369-credential-socket\") pod \"e2e-ait-50ff7514-openshift-default-679cf6446f-4z59k\" (UID: \"33935040-0f5f-43db-91c7-cae4d0e37369\") " pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-4z59k"
Oct 30 22:06:50.045073 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:50.044924    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4rks\" (UniqueName: \"kubernetes.io/projected/33935040-0f5f-43db-91c7-cae4d0e37369-kube-api-access-t4rks\") pod \"e2e-ait-50ff7514-openshift-default-679cf6446f-4z59k\" (UID: \"33935040-0f5f-43db-91c7-cae4d0e37369\") " pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-4z59k"
Oct 30 22:06:50.045073 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:50.044981    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33935040-0f5f-43db-91c7-cae4d0e37369-istio-envoy\") pod \"e2e-ait-50ff7514-openshift-default-679cf6446f-4z59k\" (UID: \"33935040-0f5f-43db-91c7-cae4d0e37369\") " pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-4z59k"
Oct 30 22:06:50.045241 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:50.045148    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33935040-0f5f-43db-91c7-cae4d0e37369-istio-podinfo\") pod \"e2e-ait-50ff7514-openshift-default-679cf6446f-4z59k\" (UID: \"33935040-0f5f-43db-91c7-cae4d0e37369\") " pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-4z59k"
Oct 30 22:06:50.045241 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:50.045182    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33935040-0f5f-43db-91c7-cae4d0e37369-istio-data\") pod \"e2e-ait-50ff7514-openshift-default-679cf6446f-4z59k\" (UID: \"33935040-0f5f-43db-91c7-cae4d0e37369\") " pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-4z59k"
Oct 30 22:06:50.045348 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:50.045253    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33935040-0f5f-43db-91c7-cae4d0e37369-workload-certs\") pod \"e2e-ait-50ff7514-openshift-default-679cf6446f-4z59k\" (UID: \"33935040-0f5f-43db-91c7-cae4d0e37369\") " pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-4z59k"
Oct 30 22:06:50.045348 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:50.045295    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33935040-0f5f-43db-91c7-cae4d0e37369-workload-socket\") pod \"e2e-ait-50ff7514-openshift-default-679cf6446f-4z59k\" (UID: \"33935040-0f5f-43db-91c7-cae4d0e37369\") " pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-4z59k"
Oct 30 22:06:50.045348 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:50.045299    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33935040-0f5f-43db-91c7-cae4d0e37369-credential-socket\") pod \"e2e-ait-50ff7514-openshift-default-679cf6446f-4z59k\" (UID: \"33935040-0f5f-43db-91c7-cae4d0e37369\") " pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-4z59k"
Oct 30 22:06:50.045507 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:50.045451    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33935040-0f5f-43db-91c7-cae4d0e37369-istiod-ca-cert\") pod \"e2e-ait-50ff7514-openshift-default-679cf6446f-4z59k\" (UID: \"33935040-0f5f-43db-91c7-cae4d0e37369\") " pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-4z59k"
Oct 30 22:06:50.045598 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:50.045543    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33935040-0f5f-43db-91c7-cae4d0e37369-istio-data\") pod \"e2e-ait-50ff7514-openshift-default-679cf6446f-4z59k\" (UID: \"33935040-0f5f-43db-91c7-cae4d0e37369\") " pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-4z59k"
Oct 30 22:06:50.045662 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:50.045629    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33935040-0f5f-43db-91c7-cae4d0e37369-workload-socket\") pod \"e2e-ait-50ff7514-openshift-default-679cf6446f-4z59k\" (UID: \"33935040-0f5f-43db-91c7-cae4d0e37369\") " pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-4z59k"
Oct 30 22:06:50.045755 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:50.045710    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33935040-0f5f-43db-91c7-cae4d0e37369-workload-certs\") pod \"e2e-ait-50ff7514-openshift-default-679cf6446f-4z59k\" (UID: \"33935040-0f5f-43db-91c7-cae4d0e37369\") " pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-4z59k"
Oct 30 22:06:50.046066 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:50.046048    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33935040-0f5f-43db-91c7-cae4d0e37369-istiod-ca-cert\") pod \"e2e-ait-50ff7514-openshift-default-679cf6446f-4z59k\" (UID: \"33935040-0f5f-43db-91c7-cae4d0e37369\") " pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-4z59k"
Oct 30 22:06:50.047346 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:50.047325    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33935040-0f5f-43db-91c7-cae4d0e37369-istio-envoy\") pod \"e2e-ait-50ff7514-openshift-default-679cf6446f-4z59k\" (UID: \"33935040-0f5f-43db-91c7-cae4d0e37369\") " pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-4z59k"
Oct 30 22:06:50.047668 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:50.047649    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33935040-0f5f-43db-91c7-cae4d0e37369-istio-podinfo\") pod \"e2e-ait-50ff7514-openshift-default-679cf6446f-4z59k\" (UID: \"33935040-0f5f-43db-91c7-cae4d0e37369\") " pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-4z59k"
Oct 30 22:06:50.103829 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:50.103748    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4rks\" (UniqueName: \"kubernetes.io/projected/33935040-0f5f-43db-91c7-cae4d0e37369-kube-api-access-t4rks\") pod \"e2e-ait-50ff7514-openshift-default-679cf6446f-4z59k\" (UID: \"33935040-0f5f-43db-91c7-cae4d0e37369\") " pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-4z59k"
Oct 30 22:06:50.103829 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:50.103746    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33935040-0f5f-43db-91c7-cae4d0e37369-istio-token\") pod \"e2e-ait-50ff7514-openshift-default-679cf6446f-4z59k\" (UID: \"33935040-0f5f-43db-91c7-cae4d0e37369\") " pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-4z59k"
Oct 30 22:06:50.154899 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:50.154866    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-4z59k"
Oct 30 22:06:50.591890 ip-10-0-130-60 kubenswrapper[5061]: W1030 22:06:50.591856    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33935040_0f5f_43db_91c7_cae4d0e37369.slice/crio-98ad1ce345024b341b77cb536d6481a49d1d6858fd60e6eec53ff122b6dd3782 WatchSource:0}: Error finding container 98ad1ce345024b341b77cb536d6481a49d1d6858fd60e6eec53ff122b6dd3782: Status 404 returned error can't find the container with id 98ad1ce345024b341b77cb536d6481a49d1d6858fd60e6eec53ff122b6dd3782
Oct 30 22:06:50.593234 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:50.593207    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-4z59k"]
Oct 30 22:06:50.594829 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:50.594796    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:06:50.594919 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:50.594860    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:06:50.594919 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:50.594889    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:06:50.709818 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:50.709778    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-4z59k" event={"ID":"33935040-0f5f-43db-91c7-cae4d0e37369","Type":"ContainerStarted","Data":"1dcfd238a6bc3bbfc4added3d6aa777a20b257613c6764f85e7a94bfda722b20"}
Oct 30 22:06:50.709978 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:50.709844    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-4z59k" event={"ID":"33935040-0f5f-43db-91c7-cae4d0e37369","Type":"ContainerStarted","Data":"98ad1ce345024b341b77cb536d6481a49d1d6858fd60e6eec53ff122b6dd3782"}
Oct 30 22:06:50.827089 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:50.827026    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-4z59k" podStartSLOduration=1.827007598 podStartE2EDuration="1.827007598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:06:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:50.821386696 +0000 UTC m=+1464.434156459" watchObservedRunningTime="2026-10-30 22:06:50.827007598 +0000 UTC m=+1464.439777357"
Oct 30 22:06:51.155046 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:51.155011    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-4z59k"
Oct 30 22:06:51.156501 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:51.156474    5061 patch_prober.go:29] interesting pod/e2e-ait-50ff7514-openshift-default-679cf6446f-4z59k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Oct 30 22:06:51.156681 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:51.156541    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-4z59k" podUID="33935040-0f5f-43db-91c7-cae4d0e37369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Oct 30 22:06:52.155889 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:52.155854    5061 patch_prober.go:29] interesting pod/e2e-ait-50ff7514-openshift-default-679cf6446f-4z59k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Oct 30 22:06:52.156282 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:52.155918    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-4z59k" podUID="33935040-0f5f-43db-91c7-cae4d0e37369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Oct 30 22:06:53.155973 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:53.155930    5061 patch_prober.go:29] interesting pod/e2e-ait-50ff7514-openshift-default-679cf6446f-4z59k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Oct 30 22:06:53.156362 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:53.155994    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-4z59k" podUID="33935040-0f5f-43db-91c7-cae4d0e37369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Oct 30 22:06:54.155953 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:54.155922    5061 patch_prober.go:29] interesting pod/e2e-ait-50ff7514-openshift-default-679cf6446f-4z59k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Oct 30 22:06:54.156135 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:54.155997    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-4z59k" podUID="33935040-0f5f-43db-91c7-cae4d0e37369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Oct 30 22:06:54.668907 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:06:54.668869    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 22:06:55.155781 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:55.155747    5061 patch_prober.go:29] interesting pod/e2e-ait-50ff7514-openshift-default-679cf6446f-4z59k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Oct 30 22:06:55.156137 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:55.155808    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-4z59k" podUID="33935040-0f5f-43db-91c7-cae4d0e37369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Oct 30 22:06:56.155692 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:56.155651    5061 patch_prober.go:29] interesting pod/e2e-ait-50ff7514-openshift-default-679cf6446f-4z59k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Oct 30 22:06:56.155903 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:56.155712    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-4z59k" podUID="33935040-0f5f-43db-91c7-cae4d0e37369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Oct 30 22:06:57.323501 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:57.323466    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-4z59k"
Oct 30 22:06:57.324002 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:57.323870    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-4z59k"
Oct 30 22:06:57.325232 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:06:57.325206    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-4z59k"
Oct 30 22:07:09.706183 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:09.706100    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-b3084d7f-openshift-default-5b87dddbbf-5h4fs"]
Oct 30 22:07:09.706666 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:09.706442    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-b3084d7f-openshift-default-5b87dddbbf-5h4fs" podUID="14ca670b-dd10-4d17-99d0-b1497d64da96" containerName="istio-proxy" containerID="cri-o://248c7bb3000f571b2427ea712d0efd834e643fc9d96fb5f8d21a6b2f144a8dc7" gracePeriod=30
Oct 30 22:07:14.968679 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:14.968583    5061 generic.go:356] "Generic (PLEG): container finished" podID="14ca670b-dd10-4d17-99d0-b1497d64da96" containerID="248c7bb3000f571b2427ea712d0efd834e643fc9d96fb5f8d21a6b2f144a8dc7" exitCode=0
Oct 30 22:07:14.969128 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:14.968788    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b3084d7f-openshift-default-5b87dddbbf-5h4fs" event={"ID":"14ca670b-dd10-4d17-99d0-b1497d64da96","Type":"ContainerDied","Data":"248c7bb3000f571b2427ea712d0efd834e643fc9d96fb5f8d21a6b2f144a8dc7"}
Oct 30 22:07:14.969128 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:14.968859    5061 scope.go:122] "RemoveContainer" containerID="e265965f3aa420f19497d908d456f7ab0c9b417c759ff5030ffe22eb68186dad"
Oct 30 22:07:15.030015 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:15.029984    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b3084d7f-openshift-default-5b87dddbbf-5h4fs"
Oct 30 22:07:15.151544 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:15.151507    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14ca670b-dd10-4d17-99d0-b1497d64da96-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14ca670b-dd10-4d17-99d0-b1497d64da96-workload-certs\") pod \"14ca670b-dd10-4d17-99d0-b1497d64da96\" (UID: \"14ca670b-dd10-4d17-99d0-b1497d64da96\") "
Oct 30 22:07:15.151744 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:15.151640    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14ca670b-dd10-4d17-99d0-b1497d64da96-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14ca670b-dd10-4d17-99d0-b1497d64da96-workload-socket\") pod \"14ca670b-dd10-4d17-99d0-b1497d64da96\" (UID: \"14ca670b-dd10-4d17-99d0-b1497d64da96\") "
Oct 30 22:07:15.151744 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:15.151707    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14ca670b-dd10-4d17-99d0-b1497d64da96-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14ca670b-dd10-4d17-99d0-b1497d64da96-istio-data\") pod \"14ca670b-dd10-4d17-99d0-b1497d64da96\" (UID: \"14ca670b-dd10-4d17-99d0-b1497d64da96\") "
Oct 30 22:07:15.151874 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:15.151758    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14ca670b-dd10-4d17-99d0-b1497d64da96-kube-api-access-7bht8\" (UniqueName: \"kubernetes.io/projected/14ca670b-dd10-4d17-99d0-b1497d64da96-kube-api-access-7bht8\") pod \"14ca670b-dd10-4d17-99d0-b1497d64da96\" (UID: \"14ca670b-dd10-4d17-99d0-b1497d64da96\") "
Oct 30 22:07:15.151874 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:15.151818    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14ca670b-dd10-4d17-99d0-b1497d64da96-workload-certs" pod "14ca670b-dd10-4d17-99d0-b1497d64da96" (UID: "14ca670b-dd10-4d17-99d0-b1497d64da96"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:07:15.151963 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:15.151913    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14ca670b-dd10-4d17-99d0-b1497d64da96-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14ca670b-dd10-4d17-99d0-b1497d64da96-istio-envoy\") pod \"14ca670b-dd10-4d17-99d0-b1497d64da96\" (UID: \"14ca670b-dd10-4d17-99d0-b1497d64da96\") "
Oct 30 22:07:15.151963 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:15.151938    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14ca670b-dd10-4d17-99d0-b1497d64da96-workload-socket" pod "14ca670b-dd10-4d17-99d0-b1497d64da96" (UID: "14ca670b-dd10-4d17-99d0-b1497d64da96"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:07:15.152027 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:15.151971    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/14ca670b-dd10-4d17-99d0-b1497d64da96-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14ca670b-dd10-4d17-99d0-b1497d64da96-istio-podinfo\") pod \"14ca670b-dd10-4d17-99d0-b1497d64da96\" (UID: \"14ca670b-dd10-4d17-99d0-b1497d64da96\") "
Oct 30 22:07:15.152027 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:15.151990    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14ca670b-dd10-4d17-99d0-b1497d64da96-istio-data" pod "14ca670b-dd10-4d17-99d0-b1497d64da96" (UID: "14ca670b-dd10-4d17-99d0-b1497d64da96"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:07:15.152027 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:15.152005    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14ca670b-dd10-4d17-99d0-b1497d64da96-istio-token\" (UniqueName: \"kubernetes.io/projected/14ca670b-dd10-4d17-99d0-b1497d64da96-istio-token\") pod \"14ca670b-dd10-4d17-99d0-b1497d64da96\" (UID: \"14ca670b-dd10-4d17-99d0-b1497d64da96\") "
Oct 30 22:07:15.152137 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:15.152061    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14ca670b-dd10-4d17-99d0-b1497d64da96-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14ca670b-dd10-4d17-99d0-b1497d64da96-credential-socket\") pod \"14ca670b-dd10-4d17-99d0-b1497d64da96\" (UID: \"14ca670b-dd10-4d17-99d0-b1497d64da96\") "
Oct 30 22:07:15.152137 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:15.152101    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14ca670b-dd10-4d17-99d0-b1497d64da96-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14ca670b-dd10-4d17-99d0-b1497d64da96-istiod-ca-cert\") pod \"14ca670b-dd10-4d17-99d0-b1497d64da96\" (UID: \"14ca670b-dd10-4d17-99d0-b1497d64da96\") "
Oct 30 22:07:15.152397 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:15.152366    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14ca670b-dd10-4d17-99d0-b1497d64da96-credential-socket" pod "14ca670b-dd10-4d17-99d0-b1497d64da96" (UID: "14ca670b-dd10-4d17-99d0-b1497d64da96"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:07:15.152542 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:15.152495    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14ca670b-dd10-4d17-99d0-b1497d64da96-istiod-ca-cert" pod "14ca670b-dd10-4d17-99d0-b1497d64da96" (UID: "14ca670b-dd10-4d17-99d0-b1497d64da96"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:07:15.153113 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:15.153089    5061 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14ca670b-dd10-4d17-99d0-b1497d64da96-istio-data\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:07:15.153177 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:15.153122    5061 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14ca670b-dd10-4d17-99d0-b1497d64da96-credential-socket\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:07:15.153177 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:15.153140    5061 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14ca670b-dd10-4d17-99d0-b1497d64da96-istiod-ca-cert\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:07:15.153177 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:15.153156    5061 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14ca670b-dd10-4d17-99d0-b1497d64da96-workload-certs\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:07:15.153177 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:15.153171    5061 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14ca670b-dd10-4d17-99d0-b1497d64da96-workload-socket\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:07:15.154064 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:15.154041    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/14ca670b-dd10-4d17-99d0-b1497d64da96-istio-podinfo" pod "14ca670b-dd10-4d17-99d0-b1497d64da96" (UID: "14ca670b-dd10-4d17-99d0-b1497d64da96"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:07:15.154265 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:15.154245    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14ca670b-dd10-4d17-99d0-b1497d64da96-kube-api-access-7bht8" pod "14ca670b-dd10-4d17-99d0-b1497d64da96" (UID: "14ca670b-dd10-4d17-99d0-b1497d64da96"). InnerVolumeSpecName "kube-api-access-7bht8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:07:15.154567 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:15.154508    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14ca670b-dd10-4d17-99d0-b1497d64da96-istio-envoy" pod "14ca670b-dd10-4d17-99d0-b1497d64da96" (UID: "14ca670b-dd10-4d17-99d0-b1497d64da96"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:07:15.154649 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:15.154578    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14ca670b-dd10-4d17-99d0-b1497d64da96-istio-token" pod "14ca670b-dd10-4d17-99d0-b1497d64da96" (UID: "14ca670b-dd10-4d17-99d0-b1497d64da96"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:07:15.255450 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:15.255360    5061 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14ca670b-dd10-4d17-99d0-b1497d64da96-istio-envoy\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:07:15.255450 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:15.255389    5061 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14ca670b-dd10-4d17-99d0-b1497d64da96-istio-podinfo\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:07:15.255450 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:15.255398    5061 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14ca670b-dd10-4d17-99d0-b1497d64da96-istio-token\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:07:15.255450 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:15.255407    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bht8\" (UniqueName: \"kubernetes.io/projected/14ca670b-dd10-4d17-99d0-b1497d64da96-kube-api-access-7bht8\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:07:15.979990 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:15.979957    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b3084d7f-openshift-default-5b87dddbbf-5h4fs" event={"ID":"14ca670b-dd10-4d17-99d0-b1497d64da96","Type":"ContainerDied","Data":"dbc4a930f78257258e4bfc7dea69ba0dbba4636566f2b4b1ad0763686f9c9117"}
Oct 30 22:07:15.980363 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:15.980013    5061 scope.go:122] "RemoveContainer" containerID="248c7bb3000f571b2427ea712d0efd834e643fc9d96fb5f8d21a6b2f144a8dc7"
Oct 30 22:07:15.980363 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:15.980089    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b3084d7f-openshift-default-5b87dddbbf-5h4fs"
Oct 30 22:07:16.188583 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:16.188548    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-b3084d7f-openshift-default-5b87dddbbf-5h4fs"]
Oct 30 22:07:16.235501 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:16.235431    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-b3084d7f-openshift-default-5b87dddbbf-5h4fs"]
Oct 30 22:07:16.533118 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:16.533037    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14ca670b-dd10-4d17-99d0-b1497d64da96" path="/var/lib/kubelet/pods/14ca670b-dd10-4d17-99d0-b1497d64da96/volumes"
Oct 30 22:07:19.504992 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:19.504957    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-13297892-66644cbc47-k8xjl"]
Oct 30 22:07:19.505998 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:19.505973    5061 cpu_manager.go:414] "containerMap: removing container" podUID="14ca670b-dd10-4d17-99d0-b1497d64da96" containerName="istio-proxy"
Oct 30 22:07:19.505998 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:19.505997    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14ca670b-dd10-4d17-99d0-b1497d64da96" containerName="istio-proxy"
Oct 30 22:07:19.506230 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:19.506214    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="14ca670b-dd10-4d17-99d0-b1497d64da96" containerName="istio-proxy"
Oct 30 22:07:19.506303 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:19.506236    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="14ca670b-dd10-4d17-99d0-b1497d64da96" containerName="istio-proxy"
Oct 30 22:07:19.510831 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:19.510783    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-13297892-66644cbc47-k8xjl"
Oct 30 22:07:19.526989 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:19.526945    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a339a53c-1e2e-45e3-af4a-1538cc2b7e00-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-13297892-66644cbc47-k8xjl\" (UID: \"a339a53c-1e2e-45e3-af4a-1538cc2b7e00\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-13297892-66644cbc47-k8xjl"
Oct 30 22:07:19.527248 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:19.527223    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9k2wb\" (UniqueName: \"kubernetes.io/projected/a339a53c-1e2e-45e3-af4a-1538cc2b7e00-kube-api-access-9k2wb\") pod \"payload-pre-processing-e2e-migrate-13297892-66644cbc47-k8xjl\" (UID: \"a339a53c-1e2e-45e3-af4a-1538cc2b7e00\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-13297892-66644cbc47-k8xjl"
Oct 30 22:07:19.570961 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:19.570920    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-13297892-66644cbc47-k8xjl"]
Oct 30 22:07:19.675611 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:19.637732    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a339a53c-1e2e-45e3-af4a-1538cc2b7e00-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-13297892-66644cbc47-k8xjl\" (UID: \"a339a53c-1e2e-45e3-af4a-1538cc2b7e00\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-13297892-66644cbc47-k8xjl"
Oct 30 22:07:19.682971 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:19.682938    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9k2wb\" (UniqueName: \"kubernetes.io/projected/a339a53c-1e2e-45e3-af4a-1538cc2b7e00-kube-api-access-9k2wb\") pod \"payload-pre-processing-e2e-migrate-13297892-66644cbc47-k8xjl\" (UID: \"a339a53c-1e2e-45e3-af4a-1538cc2b7e00\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-13297892-66644cbc47-k8xjl"
Oct 30 22:07:19.683367 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:19.641831    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-13297892-dockercfg-9dq84\""
Oct 30 22:07:19.683454 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:19.641653    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-13297892\""
Oct 30 22:07:19.683784 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:19.643883    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a339a53c-1e2e-45e3-af4a-1538cc2b7e00-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-13297892-66644cbc47-k8xjl\" (UID: \"a339a53c-1e2e-45e3-af4a-1538cc2b7e00\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-13297892-66644cbc47-k8xjl"
Oct 30 22:07:19.740969 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:19.740933    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9k2wb\" (UniqueName: \"kubernetes.io/projected/a339a53c-1e2e-45e3-af4a-1538cc2b7e00-kube-api-access-9k2wb\") pod \"payload-pre-processing-e2e-migrate-13297892-66644cbc47-k8xjl\" (UID: \"a339a53c-1e2e-45e3-af4a-1538cc2b7e00\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-13297892-66644cbc47-k8xjl"
Oct 30 22:07:19.831044 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:19.831005    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-13297892-66644cbc47-k8xjl"
Oct 30 22:07:20.250415 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:20.250321    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-13297892-66644cbc47-k8xjl"]
Oct 30 22:07:20.250587 ip-10-0-130-60 kubenswrapper[5061]: W1030 22:07:20.250477    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda339a53c_1e2e_45e3_af4a_1538cc2b7e00.slice/crio-645d35f5da50185616427c1768b0bb86ee5c2bcd9bc8f4eb331923f7862715ee WatchSource:0}: Error finding container 645d35f5da50185616427c1768b0bb86ee5c2bcd9bc8f4eb331923f7862715ee: Status 404 returned error can't find the container with id 645d35f5da50185616427c1768b0bb86ee5c2bcd9bc8f4eb331923f7862715ee
Oct 30 22:07:21.045592 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:21.045550    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-13297892-66644cbc47-k8xjl" event={"ID":"a339a53c-1e2e-45e3-af4a-1538cc2b7e00","Type":"ContainerStarted","Data":"a22ae8c774b95b466271922f96492e4ac0efdc143c5f480a17de7f68fc620190"}
Oct 30 22:07:21.046084 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:21.045647    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-13297892-66644cbc47-k8xjl"
Oct 30 22:07:21.046084 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:21.045668    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-13297892-66644cbc47-k8xjl" event={"ID":"a339a53c-1e2e-45e3-af4a-1538cc2b7e00","Type":"ContainerStarted","Data":"645d35f5da50185616427c1768b0bb86ee5c2bcd9bc8f4eb331923f7862715ee"}
Oct 30 22:07:21.162132 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:21.162078    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-13297892-66644cbc47-k8xjl" podStartSLOduration=2.162062752 podStartE2EDuration="2.162062752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:07:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:21.159386531 +0000 UTC m=+1494.772156293" watchObservedRunningTime="2026-10-30 22:07:21.162062752 +0000 UTC m=+1494.774832509"
Oct 30 22:07:24.059244 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:24.059211    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2197e210-58t8k"]
Oct 30 22:07:24.105456 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:24.105423    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2197e210-58t8k"]
Oct 30 22:07:24.531382 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:24.531301    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8002160f-c9be-4542-9147-860ddb5652ca" path="/var/lib/kubelet/pods/8002160f-c9be-4542-9147-860ddb5652ca/volumes"
Oct 30 22:07:24.669276 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:07:24.669236    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 22:07:26.743209 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:26.743182    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-sclrt_5e74071e-0462-42b5-a05c-d90bb638478a/console-operator/2.log"
Oct 30 22:07:26.745230 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:26.745198    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-sclrt_5e74071e-0462-42b5-a05c-d90bb638478a/console-operator/2.log"
Oct 30 22:07:32.057102 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:32.057068    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-13297892-66644cbc47-k8xjl"
Oct 30 22:07:35.188849 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:35.188771    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-554df"]
Oct 30 22:07:35.191948 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:35.191917    5061 cpu_manager.go:414] "containerMap: removing container" podUID="14ca670b-dd10-4d17-99d0-b1497d64da96" containerName="istio-proxy"
Oct 30 22:07:35.192141 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:35.192124    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14ca670b-dd10-4d17-99d0-b1497d64da96" containerName="istio-proxy"
Oct 30 22:07:35.198868 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:35.198843    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-554df"
Oct 30 22:07:35.320225 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:35.320177    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd1c8a85-674c-4888-8f28-de388047d1ca-workload-socket\") pod \"e2e-ait-50ff7514-openshift-default-679cf6446f-554df\" (UID: \"bd1c8a85-674c-4888-8f28-de388047d1ca\") " pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-554df"
Oct 30 22:07:35.320592 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:35.320560    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd1c8a85-674c-4888-8f28-de388047d1ca-istiod-ca-cert\") pod \"e2e-ait-50ff7514-openshift-default-679cf6446f-554df\" (UID: \"bd1c8a85-674c-4888-8f28-de388047d1ca\") " pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-554df"
Oct 30 22:07:35.320921 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:35.320900    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwb7s\" (UniqueName: \"kubernetes.io/projected/bd1c8a85-674c-4888-8f28-de388047d1ca-kube-api-access-wwb7s\") pod \"e2e-ait-50ff7514-openshift-default-679cf6446f-554df\" (UID: \"bd1c8a85-674c-4888-8f28-de388047d1ca\") " pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-554df"
Oct 30 22:07:35.321074 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:35.321058    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd1c8a85-674c-4888-8f28-de388047d1ca-workload-certs\") pod \"e2e-ait-50ff7514-openshift-default-679cf6446f-554df\" (UID: \"bd1c8a85-674c-4888-8f28-de388047d1ca\") " pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-554df"
Oct 30 22:07:35.376802 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:35.376768    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd1c8a85-674c-4888-8f28-de388047d1ca-istio-envoy\") pod \"e2e-ait-50ff7514-openshift-default-679cf6446f-554df\" (UID: \"bd1c8a85-674c-4888-8f28-de388047d1ca\") " pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-554df"
Oct 30 22:07:35.377195 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:35.377172    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd1c8a85-674c-4888-8f28-de388047d1ca-istio-token\") pod \"e2e-ait-50ff7514-openshift-default-679cf6446f-554df\" (UID: \"bd1c8a85-674c-4888-8f28-de388047d1ca\") " pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-554df"
Oct 30 22:07:35.377383 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:35.377366    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd1c8a85-674c-4888-8f28-de388047d1ca-istio-data\") pod \"e2e-ait-50ff7514-openshift-default-679cf6446f-554df\" (UID: \"bd1c8a85-674c-4888-8f28-de388047d1ca\") " pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-554df"
Oct 30 22:07:35.380392 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:35.380367    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd1c8a85-674c-4888-8f28-de388047d1ca-credential-socket\") pod \"e2e-ait-50ff7514-openshift-default-679cf6446f-554df\" (UID: \"bd1c8a85-674c-4888-8f28-de388047d1ca\") " pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-554df"
Oct 30 22:07:35.380597 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:35.380572    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd1c8a85-674c-4888-8f28-de388047d1ca-istio-podinfo\") pod \"e2e-ait-50ff7514-openshift-default-679cf6446f-554df\" (UID: \"bd1c8a85-674c-4888-8f28-de388047d1ca\") " pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-554df"
Oct 30 22:07:35.380951 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:35.343890    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-554df"]
Oct 30 22:07:35.482336 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:35.482240    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd1c8a85-674c-4888-8f28-de388047d1ca-credential-socket\") pod \"e2e-ait-50ff7514-openshift-default-679cf6446f-554df\" (UID: \"bd1c8a85-674c-4888-8f28-de388047d1ca\") " pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-554df"
Oct 30 22:07:35.482336 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:35.482278    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd1c8a85-674c-4888-8f28-de388047d1ca-istio-podinfo\") pod \"e2e-ait-50ff7514-openshift-default-679cf6446f-554df\" (UID: \"bd1c8a85-674c-4888-8f28-de388047d1ca\") " pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-554df"
Oct 30 22:07:35.482336 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:35.482307    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd1c8a85-674c-4888-8f28-de388047d1ca-workload-socket\") pod \"e2e-ait-50ff7514-openshift-default-679cf6446f-554df\" (UID: \"bd1c8a85-674c-4888-8f28-de388047d1ca\") " pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-554df"
Oct 30 22:07:35.482631 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:35.482382    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd1c8a85-674c-4888-8f28-de388047d1ca-istiod-ca-cert\") pod \"e2e-ait-50ff7514-openshift-default-679cf6446f-554df\" (UID: \"bd1c8a85-674c-4888-8f28-de388047d1ca\") " pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-554df"
Oct 30 22:07:35.482631 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:35.482536    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwb7s\" (UniqueName: \"kubernetes.io/projected/bd1c8a85-674c-4888-8f28-de388047d1ca-kube-api-access-wwb7s\") pod \"e2e-ait-50ff7514-openshift-default-679cf6446f-554df\" (UID: \"bd1c8a85-674c-4888-8f28-de388047d1ca\") " pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-554df"
Oct 30 22:07:35.482631 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:35.482574    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd1c8a85-674c-4888-8f28-de388047d1ca-workload-certs\") pod \"e2e-ait-50ff7514-openshift-default-679cf6446f-554df\" (UID: \"bd1c8a85-674c-4888-8f28-de388047d1ca\") " pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-554df"
Oct 30 22:07:35.482631 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:35.482608    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd1c8a85-674c-4888-8f28-de388047d1ca-istio-envoy\") pod \"e2e-ait-50ff7514-openshift-default-679cf6446f-554df\" (UID: \"bd1c8a85-674c-4888-8f28-de388047d1ca\") " pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-554df"
Oct 30 22:07:35.482846 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:35.482655    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd1c8a85-674c-4888-8f28-de388047d1ca-istio-token\") pod \"e2e-ait-50ff7514-openshift-default-679cf6446f-554df\" (UID: \"bd1c8a85-674c-4888-8f28-de388047d1ca\") " pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-554df"
Oct 30 22:07:35.482846 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:35.482695    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd1c8a85-674c-4888-8f28-de388047d1ca-istio-data\") pod \"e2e-ait-50ff7514-openshift-default-679cf6446f-554df\" (UID: \"bd1c8a85-674c-4888-8f28-de388047d1ca\") " pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-554df"
Oct 30 22:07:35.482846 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:35.482728    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd1c8a85-674c-4888-8f28-de388047d1ca-credential-socket\") pod \"e2e-ait-50ff7514-openshift-default-679cf6446f-554df\" (UID: \"bd1c8a85-674c-4888-8f28-de388047d1ca\") " pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-554df"
Oct 30 22:07:35.482846 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:35.482803    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd1c8a85-674c-4888-8f28-de388047d1ca-workload-socket\") pod \"e2e-ait-50ff7514-openshift-default-679cf6446f-554df\" (UID: \"bd1c8a85-674c-4888-8f28-de388047d1ca\") " pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-554df"
Oct 30 22:07:35.483036 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:35.482889    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd1c8a85-674c-4888-8f28-de388047d1ca-workload-certs\") pod \"e2e-ait-50ff7514-openshift-default-679cf6446f-554df\" (UID: \"bd1c8a85-674c-4888-8f28-de388047d1ca\") " pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-554df"
Oct 30 22:07:35.483153 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:35.483134    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd1c8a85-674c-4888-8f28-de388047d1ca-istiod-ca-cert\") pod \"e2e-ait-50ff7514-openshift-default-679cf6446f-554df\" (UID: \"bd1c8a85-674c-4888-8f28-de388047d1ca\") " pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-554df"
Oct 30 22:07:35.483199 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:35.483147    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd1c8a85-674c-4888-8f28-de388047d1ca-istio-data\") pod \"e2e-ait-50ff7514-openshift-default-679cf6446f-554df\" (UID: \"bd1c8a85-674c-4888-8f28-de388047d1ca\") " pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-554df"
Oct 30 22:07:35.484951 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:35.484925    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd1c8a85-674c-4888-8f28-de388047d1ca-istio-envoy\") pod \"e2e-ait-50ff7514-openshift-default-679cf6446f-554df\" (UID: \"bd1c8a85-674c-4888-8f28-de388047d1ca\") " pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-554df"
Oct 30 22:07:35.485063 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:35.484962    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd1c8a85-674c-4888-8f28-de388047d1ca-istio-podinfo\") pod \"e2e-ait-50ff7514-openshift-default-679cf6446f-554df\" (UID: \"bd1c8a85-674c-4888-8f28-de388047d1ca\") " pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-554df"
Oct 30 22:07:35.542630 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:35.542599    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd1c8a85-674c-4888-8f28-de388047d1ca-istio-token\") pod \"e2e-ait-50ff7514-openshift-default-679cf6446f-554df\" (UID: \"bd1c8a85-674c-4888-8f28-de388047d1ca\") " pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-554df"
Oct 30 22:07:35.542819 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:35.542795    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwb7s\" (UniqueName: \"kubernetes.io/projected/bd1c8a85-674c-4888-8f28-de388047d1ca-kube-api-access-wwb7s\") pod \"e2e-ait-50ff7514-openshift-default-679cf6446f-554df\" (UID: \"bd1c8a85-674c-4888-8f28-de388047d1ca\") " pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-554df"
Oct 30 22:07:35.815737 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:35.815706    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-554df"
Oct 30 22:07:36.248789 ip-10-0-130-60 kubenswrapper[5061]: W1030 22:07:36.248751    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd1c8a85_674c_4888_8f28_de388047d1ca.slice/crio-8ba69f6e2db1965c65e85e90c9929cf59a3f5dc0ad67593c1f87555ef92a29e7 WatchSource:0}: Error finding container 8ba69f6e2db1965c65e85e90c9929cf59a3f5dc0ad67593c1f87555ef92a29e7: Status 404 returned error can't find the container with id 8ba69f6e2db1965c65e85e90c9929cf59a3f5dc0ad67593c1f87555ef92a29e7
Oct 30 22:07:36.257029 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:36.256961    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:07:36.257148 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:36.257100    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:07:36.257217 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:36.257150    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:07:36.258681 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:36.258637    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-554df"]
Oct 30 22:07:37.241727 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:37.241687    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-554df" event={"ID":"bd1c8a85-674c-4888-8f28-de388047d1ca","Type":"ContainerStarted","Data":"33b04bfb1165d83967a60df5ec6458bde9e20ea142b7cd8f5db86265811b6142"}
Oct 30 22:07:37.241921 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:37.241744    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-554df" event={"ID":"bd1c8a85-674c-4888-8f28-de388047d1ca","Type":"ContainerStarted","Data":"8ba69f6e2db1965c65e85e90c9929cf59a3f5dc0ad67593c1f87555ef92a29e7"}
Oct 30 22:07:37.472944 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:37.472888    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-554df" podStartSLOduration=2.472873391 podStartE2EDuration="2.472873391s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:07:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:37.419070083 +0000 UTC m=+1511.031839869" watchObservedRunningTime="2026-10-30 22:07:37.472873391 +0000 UTC m=+1511.085643149"
Oct 30 22:07:37.833131 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:37.816219    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-554df"
Oct 30 22:07:37.833131 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:37.832674    5061 patch_prober.go:29] interesting pod/e2e-ait-50ff7514-openshift-default-679cf6446f-554df container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body=
Oct 30 22:07:37.833131 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:37.832722    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-554df" podUID="bd1c8a85-674c-4888-8f28-de388047d1ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused"
Oct 30 22:07:38.577254 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:38.577218    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-4z59k"]
Oct 30 22:07:38.577669 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:38.577495    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-4z59k" podUID="33935040-0f5f-43db-91c7-cae4d0e37369" containerName="istio-proxy" containerID="cri-o://1dcfd238a6bc3bbfc4added3d6aa777a20b257613c6764f85e7a94bfda722b20" gracePeriod=30
Oct 30 22:07:38.601402 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:38.601362    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-554df"]
Oct 30 22:07:38.863135 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:38.863038    5061 patch_prober.go:29] interesting pod/e2e-ait-50ff7514-openshift-default-679cf6446f-554df container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body=
Oct 30 22:07:38.863135 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:38.863104    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-554df" podUID="bd1c8a85-674c-4888-8f28-de388047d1ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused"
Oct 30 22:07:39.280661 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:39.278679    5061 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-554df" secret="" err="secret \"e2e-ait-50ff7514-openshift-default-dockercfg-g859w\" not found"
Oct 30 22:07:39.830596 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:39.826953    5061 patch_prober.go:29] interesting pod/e2e-ait-50ff7514-openshift-default-679cf6446f-554df container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body=
Oct 30 22:07:39.830596 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:39.827018    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-554df" podUID="bd1c8a85-674c-4888-8f28-de388047d1ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused"
Oct 30 22:07:40.285714 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:40.285591    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-554df" podUID="bd1c8a85-674c-4888-8f28-de388047d1ca" containerName="istio-proxy" containerID="cri-o://33b04bfb1165d83967a60df5ec6458bde9e20ea142b7cd8f5db86265811b6142" gracePeriod=30
Oct 30 22:07:42.326037 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:42.325997    5061 patch_prober.go:29] interesting pod/e2e-ait-50ff7514-openshift-default-679cf6446f-4z59k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:07:42.326461 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:42.326065    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-4z59k" podUID="33935040-0f5f-43db-91c7-cae4d0e37369" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:07:43.861026 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:43.860964    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-4z59k"
Oct 30 22:07:43.928169 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:43.928143    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33935040-0f5f-43db-91c7-cae4d0e37369-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33935040-0f5f-43db-91c7-cae4d0e37369-istiod-ca-cert\") pod \"33935040-0f5f-43db-91c7-cae4d0e37369\" (UID: \"33935040-0f5f-43db-91c7-cae4d0e37369\") "
Oct 30 22:07:43.928329 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:43.928317    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33935040-0f5f-43db-91c7-cae4d0e37369-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33935040-0f5f-43db-91c7-cae4d0e37369-istio-data\") pod \"33935040-0f5f-43db-91c7-cae4d0e37369\" (UID: \"33935040-0f5f-43db-91c7-cae4d0e37369\") "
Oct 30 22:07:43.928427 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:43.928415    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/33935040-0f5f-43db-91c7-cae4d0e37369-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33935040-0f5f-43db-91c7-cae4d0e37369-istio-podinfo\") pod \"33935040-0f5f-43db-91c7-cae4d0e37369\" (UID: \"33935040-0f5f-43db-91c7-cae4d0e37369\") "
Oct 30 22:07:43.928471 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:43.928452    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33935040-0f5f-43db-91c7-cae4d0e37369-istio-token\" (UniqueName: \"kubernetes.io/projected/33935040-0f5f-43db-91c7-cae4d0e37369-istio-token\") pod \"33935040-0f5f-43db-91c7-cae4d0e37369\" (UID: \"33935040-0f5f-43db-91c7-cae4d0e37369\") "
Oct 30 22:07:43.928521 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:43.928472    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33935040-0f5f-43db-91c7-cae4d0e37369-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33935040-0f5f-43db-91c7-cae4d0e37369-istio-envoy\") pod \"33935040-0f5f-43db-91c7-cae4d0e37369\" (UID: \"33935040-0f5f-43db-91c7-cae4d0e37369\") "
Oct 30 22:07:43.928521 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:43.928493    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33935040-0f5f-43db-91c7-cae4d0e37369-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33935040-0f5f-43db-91c7-cae4d0e37369-workload-certs\") pod \"33935040-0f5f-43db-91c7-cae4d0e37369\" (UID: \"33935040-0f5f-43db-91c7-cae4d0e37369\") "
Oct 30 22:07:43.928521 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:43.928496    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33935040-0f5f-43db-91c7-cae4d0e37369-istiod-ca-cert" pod "33935040-0f5f-43db-91c7-cae4d0e37369" (UID: "33935040-0f5f-43db-91c7-cae4d0e37369"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:07:43.928689 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:43.928626    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33935040-0f5f-43db-91c7-cae4d0e37369-istio-data" pod "33935040-0f5f-43db-91c7-cae4d0e37369" (UID: "33935040-0f5f-43db-91c7-cae4d0e37369"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:07:43.928782 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:43.928757    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33935040-0f5f-43db-91c7-cae4d0e37369-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33935040-0f5f-43db-91c7-cae4d0e37369-credential-socket\") pod \"33935040-0f5f-43db-91c7-cae4d0e37369\" (UID: \"33935040-0f5f-43db-91c7-cae4d0e37369\") "
Oct 30 22:07:43.928848 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:43.928784    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33935040-0f5f-43db-91c7-cae4d0e37369-workload-certs" pod "33935040-0f5f-43db-91c7-cae4d0e37369" (UID: "33935040-0f5f-43db-91c7-cae4d0e37369"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:07:43.928848 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:43.928812    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33935040-0f5f-43db-91c7-cae4d0e37369-kube-api-access-t4rks\" (UniqueName: \"kubernetes.io/projected/33935040-0f5f-43db-91c7-cae4d0e37369-kube-api-access-t4rks\") pod \"33935040-0f5f-43db-91c7-cae4d0e37369\" (UID: \"33935040-0f5f-43db-91c7-cae4d0e37369\") "
Oct 30 22:07:43.928962 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:43.928869    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33935040-0f5f-43db-91c7-cae4d0e37369-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33935040-0f5f-43db-91c7-cae4d0e37369-workload-socket\") pod \"33935040-0f5f-43db-91c7-cae4d0e37369\" (UID: \"33935040-0f5f-43db-91c7-cae4d0e37369\") "
Oct 30 22:07:43.928962 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:43.928943    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33935040-0f5f-43db-91c7-cae4d0e37369-credential-socket" pod "33935040-0f5f-43db-91c7-cae4d0e37369" (UID: "33935040-0f5f-43db-91c7-cae4d0e37369"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:07:43.929117 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:43.929096    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33935040-0f5f-43db-91c7-cae4d0e37369-workload-socket" pod "33935040-0f5f-43db-91c7-cae4d0e37369" (UID: "33935040-0f5f-43db-91c7-cae4d0e37369"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:07:43.929552 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:43.929518    5061 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33935040-0f5f-43db-91c7-cae4d0e37369-istio-data\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:07:43.929552 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:43.929552    5061 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33935040-0f5f-43db-91c7-cae4d0e37369-workload-certs\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:07:43.929699 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:43.929564    5061 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33935040-0f5f-43db-91c7-cae4d0e37369-credential-socket\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:07:43.929699 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:43.929573    5061 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33935040-0f5f-43db-91c7-cae4d0e37369-workload-socket\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:07:43.929699 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:43.929581    5061 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33935040-0f5f-43db-91c7-cae4d0e37369-istiod-ca-cert\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:07:43.930854 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:43.930829    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33935040-0f5f-43db-91c7-cae4d0e37369-istio-envoy" pod "33935040-0f5f-43db-91c7-cae4d0e37369" (UID: "33935040-0f5f-43db-91c7-cae4d0e37369"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:07:43.930948 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:43.930869    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33935040-0f5f-43db-91c7-cae4d0e37369-kube-api-access-t4rks" pod "33935040-0f5f-43db-91c7-cae4d0e37369" (UID: "33935040-0f5f-43db-91c7-cae4d0e37369"). InnerVolumeSpecName "kube-api-access-t4rks". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:07:43.931127 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:43.931110    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/33935040-0f5f-43db-91c7-cae4d0e37369-istio-podinfo" pod "33935040-0f5f-43db-91c7-cae4d0e37369" (UID: "33935040-0f5f-43db-91c7-cae4d0e37369"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:07:43.931181 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:43.931147    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33935040-0f5f-43db-91c7-cae4d0e37369-istio-token" pod "33935040-0f5f-43db-91c7-cae4d0e37369" (UID: "33935040-0f5f-43db-91c7-cae4d0e37369"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:07:44.031282 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:44.031251    5061 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33935040-0f5f-43db-91c7-cae4d0e37369-istio-podinfo\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:07:44.031282 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:44.031279    5061 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33935040-0f5f-43db-91c7-cae4d0e37369-istio-token\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:07:44.031282 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:44.031289    5061 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33935040-0f5f-43db-91c7-cae4d0e37369-istio-envoy\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:07:44.031508 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:44.031299    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4rks\" (UniqueName: \"kubernetes.io/projected/33935040-0f5f-43db-91c7-cae4d0e37369-kube-api-access-t4rks\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:07:44.317238 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:44.317200    5061 generic.go:356] "Generic (PLEG): container finished" podID="33935040-0f5f-43db-91c7-cae4d0e37369" containerID="1dcfd238a6bc3bbfc4added3d6aa777a20b257613c6764f85e7a94bfda722b20" exitCode=0
Oct 30 22:07:44.317406 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:44.317250    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-4z59k"
Oct 30 22:07:44.317406 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:44.317280    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-4z59k" event={"ID":"33935040-0f5f-43db-91c7-cae4d0e37369","Type":"ContainerDied","Data":"1dcfd238a6bc3bbfc4added3d6aa777a20b257613c6764f85e7a94bfda722b20"}
Oct 30 22:07:44.317406 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:44.317356    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-4z59k" event={"ID":"33935040-0f5f-43db-91c7-cae4d0e37369","Type":"ContainerDied","Data":"98ad1ce345024b341b77cb536d6481a49d1d6858fd60e6eec53ff122b6dd3782"}
Oct 30 22:07:44.317406 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:44.317380    5061 scope.go:122] "RemoveContainer" containerID="1dcfd238a6bc3bbfc4added3d6aa777a20b257613c6764f85e7a94bfda722b20"
Oct 30 22:07:44.329888 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:44.329870    5061 scope.go:122] "RemoveContainer" containerID="1dcfd238a6bc3bbfc4added3d6aa777a20b257613c6764f85e7a94bfda722b20"
Oct 30 22:07:44.330120 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:07:44.330100    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1dcfd238a6bc3bbfc4added3d6aa777a20b257613c6764f85e7a94bfda722b20\": container with ID starting with 1dcfd238a6bc3bbfc4added3d6aa777a20b257613c6764f85e7a94bfda722b20 not found: ID does not exist" containerID="1dcfd238a6bc3bbfc4added3d6aa777a20b257613c6764f85e7a94bfda722b20"
Oct 30 22:07:44.330174 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:44.330128    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1dcfd238a6bc3bbfc4added3d6aa777a20b257613c6764f85e7a94bfda722b20"} err="failed to get container status \"1dcfd238a6bc3bbfc4added3d6aa777a20b257613c6764f85e7a94bfda722b20\": rpc error: code = NotFound desc = could not find container \"1dcfd238a6bc3bbfc4added3d6aa777a20b257613c6764f85e7a94bfda722b20\": container with ID starting with 1dcfd238a6bc3bbfc4added3d6aa777a20b257613c6764f85e7a94bfda722b20 not found: ID does not exist"
Oct 30 22:07:44.494080 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:44.494046    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-4z59k"]
Oct 30 22:07:44.546446 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:44.546419    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-4z59k"]
Oct 30 22:07:46.530436 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:46.530403    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33935040-0f5f-43db-91c7-cae4d0e37369" path="/var/lib/kubelet/pods/33935040-0f5f-43db-91c7-cae4d0e37369/volumes"
Oct 30 22:07:47.688299 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:47.688274    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-554df"
Oct 30 22:07:47.793388 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:47.793355    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd1c8a85-674c-4888-8f28-de388047d1ca-istio-token\" (UniqueName: \"kubernetes.io/projected/bd1c8a85-674c-4888-8f28-de388047d1ca-istio-token\") pod \"bd1c8a85-674c-4888-8f28-de388047d1ca\" (UID: \"bd1c8a85-674c-4888-8f28-de388047d1ca\") "
Oct 30 22:07:47.793586 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:47.793427    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bd1c8a85-674c-4888-8f28-de388047d1ca-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd1c8a85-674c-4888-8f28-de388047d1ca-istio-podinfo\") pod \"bd1c8a85-674c-4888-8f28-de388047d1ca\" (UID: \"bd1c8a85-674c-4888-8f28-de388047d1ca\") "
Oct 30 22:07:47.793586 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:47.793453    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd1c8a85-674c-4888-8f28-de388047d1ca-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd1c8a85-674c-4888-8f28-de388047d1ca-istio-data\") pod \"bd1c8a85-674c-4888-8f28-de388047d1ca\" (UID: \"bd1c8a85-674c-4888-8f28-de388047d1ca\") "
Oct 30 22:07:47.793586 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:47.793544    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd1c8a85-674c-4888-8f28-de388047d1ca-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd1c8a85-674c-4888-8f28-de388047d1ca-workload-certs\") pod \"bd1c8a85-674c-4888-8f28-de388047d1ca\" (UID: \"bd1c8a85-674c-4888-8f28-de388047d1ca\") "
Oct 30 22:07:47.793851 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:47.793589    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd1c8a85-674c-4888-8f28-de388047d1ca-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd1c8a85-674c-4888-8f28-de388047d1ca-istiod-ca-cert\") pod \"bd1c8a85-674c-4888-8f28-de388047d1ca\" (UID: \"bd1c8a85-674c-4888-8f28-de388047d1ca\") "
Oct 30 22:07:47.793851 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:47.793637    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd1c8a85-674c-4888-8f28-de388047d1ca-kube-api-access-wwb7s\" (UniqueName: \"kubernetes.io/projected/bd1c8a85-674c-4888-8f28-de388047d1ca-kube-api-access-wwb7s\") pod \"bd1c8a85-674c-4888-8f28-de388047d1ca\" (UID: \"bd1c8a85-674c-4888-8f28-de388047d1ca\") "
Oct 30 22:07:47.793851 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:47.793769    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd1c8a85-674c-4888-8f28-de388047d1ca-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd1c8a85-674c-4888-8f28-de388047d1ca-istio-envoy\") pod \"bd1c8a85-674c-4888-8f28-de388047d1ca\" (UID: \"bd1c8a85-674c-4888-8f28-de388047d1ca\") "
Oct 30 22:07:47.793851 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:47.793791    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd1c8a85-674c-4888-8f28-de388047d1ca-workload-certs" pod "bd1c8a85-674c-4888-8f28-de388047d1ca" (UID: "bd1c8a85-674c-4888-8f28-de388047d1ca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:07:47.793851 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:47.793816    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd1c8a85-674c-4888-8f28-de388047d1ca-istio-data" pod "bd1c8a85-674c-4888-8f28-de388047d1ca" (UID: "bd1c8a85-674c-4888-8f28-de388047d1ca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:07:47.794066 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:47.793934    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd1c8a85-674c-4888-8f28-de388047d1ca-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd1c8a85-674c-4888-8f28-de388047d1ca-credential-socket\") pod \"bd1c8a85-674c-4888-8f28-de388047d1ca\" (UID: \"bd1c8a85-674c-4888-8f28-de388047d1ca\") "
Oct 30 22:07:47.794066 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:47.793984    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd1c8a85-674c-4888-8f28-de388047d1ca-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd1c8a85-674c-4888-8f28-de388047d1ca-workload-socket\") pod \"bd1c8a85-674c-4888-8f28-de388047d1ca\" (UID: \"bd1c8a85-674c-4888-8f28-de388047d1ca\") "
Oct 30 22:07:47.794309 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:47.794270    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd1c8a85-674c-4888-8f28-de388047d1ca-credential-socket" pod "bd1c8a85-674c-4888-8f28-de388047d1ca" (UID: "bd1c8a85-674c-4888-8f28-de388047d1ca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:07:47.794398 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:47.794278    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd1c8a85-674c-4888-8f28-de388047d1ca-istiod-ca-cert" pod "bd1c8a85-674c-4888-8f28-de388047d1ca" (UID: "bd1c8a85-674c-4888-8f28-de388047d1ca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:07:47.794463 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:47.794317    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd1c8a85-674c-4888-8f28-de388047d1ca-workload-socket" pod "bd1c8a85-674c-4888-8f28-de388047d1ca" (UID: "bd1c8a85-674c-4888-8f28-de388047d1ca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:07:47.794930 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:47.794906    5061 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd1c8a85-674c-4888-8f28-de388047d1ca-istio-data\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:07:47.795036 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:47.794936    5061 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd1c8a85-674c-4888-8f28-de388047d1ca-workload-certs\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:07:47.795036 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:47.794950    5061 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd1c8a85-674c-4888-8f28-de388047d1ca-istiod-ca-cert\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:07:47.795036 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:47.794965    5061 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd1c8a85-674c-4888-8f28-de388047d1ca-credential-socket\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:07:47.795036 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:47.794979    5061 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd1c8a85-674c-4888-8f28-de388047d1ca-workload-socket\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:07:47.795935 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:47.795905    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bd1c8a85-674c-4888-8f28-de388047d1ca-istio-podinfo" pod "bd1c8a85-674c-4888-8f28-de388047d1ca" (UID: "bd1c8a85-674c-4888-8f28-de388047d1ca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:07:47.796029 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:47.795974    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd1c8a85-674c-4888-8f28-de388047d1ca-kube-api-access-wwb7s" pod "bd1c8a85-674c-4888-8f28-de388047d1ca" (UID: "bd1c8a85-674c-4888-8f28-de388047d1ca"). InnerVolumeSpecName "kube-api-access-wwb7s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:07:47.796029 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:47.795996    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd1c8a85-674c-4888-8f28-de388047d1ca-istio-token" pod "bd1c8a85-674c-4888-8f28-de388047d1ca" (UID: "bd1c8a85-674c-4888-8f28-de388047d1ca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:07:47.796361 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:47.796345    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd1c8a85-674c-4888-8f28-de388047d1ca-istio-envoy" pod "bd1c8a85-674c-4888-8f28-de388047d1ca" (UID: "bd1c8a85-674c-4888-8f28-de388047d1ca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:07:47.896775 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:47.896738    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwb7s\" (UniqueName: \"kubernetes.io/projected/bd1c8a85-674c-4888-8f28-de388047d1ca-kube-api-access-wwb7s\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:07:47.896775 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:47.896769    5061 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd1c8a85-674c-4888-8f28-de388047d1ca-istio-envoy\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:07:47.896775 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:47.896780    5061 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd1c8a85-674c-4888-8f28-de388047d1ca-istio-token\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:07:47.896989 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:47.896789    5061 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd1c8a85-674c-4888-8f28-de388047d1ca-istio-podinfo\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:07:48.351081 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:48.351045    5061 generic.go:356] "Generic (PLEG): container finished" podID="bd1c8a85-674c-4888-8f28-de388047d1ca" containerID="33b04bfb1165d83967a60df5ec6458bde9e20ea142b7cd8f5db86265811b6142" exitCode=0
Oct 30 22:07:48.351257 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:48.351095    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-554df"
Oct 30 22:07:48.351257 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:48.351130    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-554df" event={"ID":"bd1c8a85-674c-4888-8f28-de388047d1ca","Type":"ContainerDied","Data":"33b04bfb1165d83967a60df5ec6458bde9e20ea142b7cd8f5db86265811b6142"}
Oct 30 22:07:48.351257 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:48.351188    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-554df" event={"ID":"bd1c8a85-674c-4888-8f28-de388047d1ca","Type":"ContainerDied","Data":"8ba69f6e2db1965c65e85e90c9929cf59a3f5dc0ad67593c1f87555ef92a29e7"}
Oct 30 22:07:48.351257 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:48.351218    5061 scope.go:122] "RemoveContainer" containerID="33b04bfb1165d83967a60df5ec6458bde9e20ea142b7cd8f5db86265811b6142"
Oct 30 22:07:48.363263 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:48.363245    5061 scope.go:122] "RemoveContainer" containerID="33b04bfb1165d83967a60df5ec6458bde9e20ea142b7cd8f5db86265811b6142"
Oct 30 22:07:48.363522 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:07:48.363498    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33b04bfb1165d83967a60df5ec6458bde9e20ea142b7cd8f5db86265811b6142\": container with ID starting with 33b04bfb1165d83967a60df5ec6458bde9e20ea142b7cd8f5db86265811b6142 not found: ID does not exist" containerID="33b04bfb1165d83967a60df5ec6458bde9e20ea142b7cd8f5db86265811b6142"
Oct 30 22:07:48.363691 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:48.363565    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33b04bfb1165d83967a60df5ec6458bde9e20ea142b7cd8f5db86265811b6142"} err="failed to get container status \"33b04bfb1165d83967a60df5ec6458bde9e20ea142b7cd8f5db86265811b6142\": rpc error: code = NotFound desc = could not find container \"33b04bfb1165d83967a60df5ec6458bde9e20ea142b7cd8f5db86265811b6142\": container with ID starting with 33b04bfb1165d83967a60df5ec6458bde9e20ea142b7cd8f5db86265811b6142 not found: ID does not exist"
Oct 30 22:07:48.530716 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:48.530679    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-554df"]
Oct 30 22:07:48.580850 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:48.580818    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-554df"]
Oct 30 22:07:50.530694 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:50.530656    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd1c8a85-674c-4888-8f28-de388047d1ca" path="/var/lib/kubelet/pods/bd1c8a85-674c-4888-8f28-de388047d1ca/volumes"
Oct 30 22:07:54.674175 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:07:54.674131    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 22:07:56.989552 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:56.974032    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-c79hw"]
Oct 30 22:07:56.989552 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:56.974348    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-c79hw" podUID="bfae7061-0326-46fd-901b-0399080c2094" containerName="istio-proxy" containerID="cri-o://888f926dda2182b4bd0ce60084acaa9343c93fc6be5e86dc479441e7cc6f50be" gracePeriod=30
Oct 30 22:07:57.067552 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:57.066655    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-wg5rk"]
Oct 30 22:07:57.067552 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:07:57.066890    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-wg5rk" podUID="025aea26-9742-4272-9249-2e916c03d53d" containerName="istio-proxy" containerID="cri-o://20f371dff9c18ed9ac092238deb6a2806e4a79f27afa197072a9fa0f4aae236c" gracePeriod=30
Oct 30 22:08:02.361000 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:08:02.360964    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-c79hw"
Oct 30 22:08:02.423657 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:08:02.423627    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-wg5rk"
Oct 30 22:08:02.492327 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:08:02.492291    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/025aea26-9742-4272-9249-2e916c03d53d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/025aea26-9742-4272-9249-2e916c03d53d-workload-certs\") pod \"025aea26-9742-4272-9249-2e916c03d53d\" (UID: \"025aea26-9742-4272-9249-2e916c03d53d\") "
Oct 30 22:08:02.492545 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:08:02.492369    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/025aea26-9742-4272-9249-2e916c03d53d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/025aea26-9742-4272-9249-2e916c03d53d-istio-envoy\") pod \"025aea26-9742-4272-9249-2e916c03d53d\" (UID: \"025aea26-9742-4272-9249-2e916c03d53d\") "
Oct 30 22:08:02.492545 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:08:02.492396    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfae7061-0326-46fd-901b-0399080c2094-kube-api-access-bxlzx\" (UniqueName: \"kubernetes.io/projected/bfae7061-0326-46fd-901b-0399080c2094-kube-api-access-bxlzx\") pod \"bfae7061-0326-46fd-901b-0399080c2094\" (UID: \"bfae7061-0326-46fd-901b-0399080c2094\") "
Oct 30 22:08:02.492545 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:08:02.492413    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfae7061-0326-46fd-901b-0399080c2094-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfae7061-0326-46fd-901b-0399080c2094-istio-envoy\") pod \"bfae7061-0326-46fd-901b-0399080c2094\" (UID: \"bfae7061-0326-46fd-901b-0399080c2094\") "
Oct 30 22:08:02.492545 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:08:02.492475    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfae7061-0326-46fd-901b-0399080c2094-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfae7061-0326-46fd-901b-0399080c2094-credential-socket\") pod \"bfae7061-0326-46fd-901b-0399080c2094\" (UID: \"bfae7061-0326-46fd-901b-0399080c2094\") "
Oct 30 22:08:02.492545 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:08:02.492509    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/025aea26-9742-4272-9249-2e916c03d53d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/025aea26-9742-4272-9249-2e916c03d53d-workload-socket\") pod \"025aea26-9742-4272-9249-2e916c03d53d\" (UID: \"025aea26-9742-4272-9249-2e916c03d53d\") "
Oct 30 22:08:02.492785 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:08:02.492587    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/025aea26-9742-4272-9249-2e916c03d53d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/025aea26-9742-4272-9249-2e916c03d53d-istio-data\") pod \"025aea26-9742-4272-9249-2e916c03d53d\" (UID: \"025aea26-9742-4272-9249-2e916c03d53d\") "
Oct 30 22:08:02.492785 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:08:02.492635    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bfae7061-0326-46fd-901b-0399080c2094-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfae7061-0326-46fd-901b-0399080c2094-istio-podinfo\") pod \"bfae7061-0326-46fd-901b-0399080c2094\" (UID: \"bfae7061-0326-46fd-901b-0399080c2094\") "
Oct 30 22:08:02.492785 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:08:02.492673    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/025aea26-9742-4272-9249-2e916c03d53d-workload-certs" pod "025aea26-9742-4272-9249-2e916c03d53d" (UID: "025aea26-9742-4272-9249-2e916c03d53d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:08:02.492935 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:08:02.492803    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/025aea26-9742-4272-9249-2e916c03d53d-istio-token\" (UniqueName: \"kubernetes.io/projected/025aea26-9742-4272-9249-2e916c03d53d-istio-token\") pod \"025aea26-9742-4272-9249-2e916c03d53d\" (UID: \"025aea26-9742-4272-9249-2e916c03d53d\") "
Oct 30 22:08:02.492935 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:08:02.492830    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfae7061-0326-46fd-901b-0399080c2094-credential-socket" pod "bfae7061-0326-46fd-901b-0399080c2094" (UID: "bfae7061-0326-46fd-901b-0399080c2094"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:08:02.492935 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:08:02.492848    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/025aea26-9742-4272-9249-2e916c03d53d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/025aea26-9742-4272-9249-2e916c03d53d-istiod-ca-cert\") pod \"025aea26-9742-4272-9249-2e916c03d53d\" (UID: \"025aea26-9742-4272-9249-2e916c03d53d\") "
Oct 30 22:08:02.492935 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:08:02.492901    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfae7061-0326-46fd-901b-0399080c2094-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfae7061-0326-46fd-901b-0399080c2094-istio-data\") pod \"bfae7061-0326-46fd-901b-0399080c2094\" (UID: \"bfae7061-0326-46fd-901b-0399080c2094\") "
Oct 30 22:08:02.493182 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:08:02.492926    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/025aea26-9742-4272-9249-2e916c03d53d-workload-socket" pod "025aea26-9742-4272-9249-2e916c03d53d" (UID: "025aea26-9742-4272-9249-2e916c03d53d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:08:02.493182 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:08:02.492963    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfae7061-0326-46fd-901b-0399080c2094-istio-token\" (UniqueName: \"kubernetes.io/projected/bfae7061-0326-46fd-901b-0399080c2094-istio-token\") pod \"bfae7061-0326-46fd-901b-0399080c2094\" (UID: \"bfae7061-0326-46fd-901b-0399080c2094\") "
Oct 30 22:08:02.493182 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:08:02.493125    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfae7061-0326-46fd-901b-0399080c2094-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfae7061-0326-46fd-901b-0399080c2094-workload-certs\") pod \"bfae7061-0326-46fd-901b-0399080c2094\" (UID: \"bfae7061-0326-46fd-901b-0399080c2094\") "
Oct 30 22:08:02.493349 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:08:02.493267    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/025aea26-9742-4272-9249-2e916c03d53d-istio-data" pod "025aea26-9742-4272-9249-2e916c03d53d" (UID: "025aea26-9742-4272-9249-2e916c03d53d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:08:02.493349 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:08:02.493302    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/025aea26-9742-4272-9249-2e916c03d53d-kube-api-access-45jmz\" (UniqueName: \"kubernetes.io/projected/025aea26-9742-4272-9249-2e916c03d53d-kube-api-access-45jmz\") pod \"025aea26-9742-4272-9249-2e916c03d53d\" (UID: \"025aea26-9742-4272-9249-2e916c03d53d\") "
Oct 30 22:08:02.493459 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:08:02.493349    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfae7061-0326-46fd-901b-0399080c2094-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfae7061-0326-46fd-901b-0399080c2094-istiod-ca-cert\") pod \"bfae7061-0326-46fd-901b-0399080c2094\" (UID: \"bfae7061-0326-46fd-901b-0399080c2094\") "
Oct 30 22:08:02.493569 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:08:02.493517    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/025aea26-9742-4272-9249-2e916c03d53d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/025aea26-9742-4272-9249-2e916c03d53d-credential-socket\") pod \"025aea26-9742-4272-9249-2e916c03d53d\" (UID: \"025aea26-9742-4272-9249-2e916c03d53d\") "
Oct 30 22:08:02.493644 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:08:02.493574    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/025aea26-9742-4272-9249-2e916c03d53d-istiod-ca-cert" pod "025aea26-9742-4272-9249-2e916c03d53d" (UID: "025aea26-9742-4272-9249-2e916c03d53d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:08:02.493754 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:08:02.493708    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfae7061-0326-46fd-901b-0399080c2094-istio-data" pod "bfae7061-0326-46fd-901b-0399080c2094" (UID: "bfae7061-0326-46fd-901b-0399080c2094"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:08:02.493875 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:08:02.493781    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfae7061-0326-46fd-901b-0399080c2094-workload-certs" pod "bfae7061-0326-46fd-901b-0399080c2094" (UID: "bfae7061-0326-46fd-901b-0399080c2094"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:08:02.493939 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:08:02.493906    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfae7061-0326-46fd-901b-0399080c2094-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfae7061-0326-46fd-901b-0399080c2094-workload-socket\") pod \"bfae7061-0326-46fd-901b-0399080c2094\" (UID: \"bfae7061-0326-46fd-901b-0399080c2094\") "
Oct 30 22:08:02.494043 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:08:02.493965    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/025aea26-9742-4272-9249-2e916c03d53d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/025aea26-9742-4272-9249-2e916c03d53d-istio-podinfo\") pod \"025aea26-9742-4272-9249-2e916c03d53d\" (UID: \"025aea26-9742-4272-9249-2e916c03d53d\") "
Oct 30 22:08:02.494209 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:08:02.494144    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfae7061-0326-46fd-901b-0399080c2094-istiod-ca-cert" pod "bfae7061-0326-46fd-901b-0399080c2094" (UID: "bfae7061-0326-46fd-901b-0399080c2094"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:08:02.495113 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:08:02.494398    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfae7061-0326-46fd-901b-0399080c2094-workload-socket" pod "bfae7061-0326-46fd-901b-0399080c2094" (UID: "bfae7061-0326-46fd-901b-0399080c2094"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:08:02.495113 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:08:02.494886    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/025aea26-9742-4272-9249-2e916c03d53d-credential-socket" pod "025aea26-9742-4272-9249-2e916c03d53d" (UID: "025aea26-9742-4272-9249-2e916c03d53d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:08:02.495113 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:08:02.494951    5061 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/025aea26-9742-4272-9249-2e916c03d53d-istiod-ca-cert\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:02.495113 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:08:02.494975    5061 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfae7061-0326-46fd-901b-0399080c2094-istio-data\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:02.495113 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:08:02.494991    5061 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfae7061-0326-46fd-901b-0399080c2094-workload-certs\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:02.495113 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:08:02.495006    5061 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfae7061-0326-46fd-901b-0399080c2094-istiod-ca-cert\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:02.495113 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:08:02.495021    5061 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfae7061-0326-46fd-901b-0399080c2094-workload-socket\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:02.495113 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:08:02.495036    5061 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/025aea26-9742-4272-9249-2e916c03d53d-workload-certs\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:02.495113 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:08:02.495052    5061 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfae7061-0326-46fd-901b-0399080c2094-credential-socket\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:02.495113 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:08:02.495066    5061 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/025aea26-9742-4272-9249-2e916c03d53d-workload-socket\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:02.495113 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:08:02.495081    5061 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/025aea26-9742-4272-9249-2e916c03d53d-istio-data\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:02.495771 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:08:02.495235    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/025aea26-9742-4272-9249-2e916c03d53d-istio-envoy" pod "025aea26-9742-4272-9249-2e916c03d53d" (UID: "025aea26-9742-4272-9249-2e916c03d53d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:08:02.496682 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:08:02.496657    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfae7061-0326-46fd-901b-0399080c2094-istio-envoy" pod "bfae7061-0326-46fd-901b-0399080c2094" (UID: "bfae7061-0326-46fd-901b-0399080c2094"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:08:02.496770 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:08:02.496686    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bfae7061-0326-46fd-901b-0399080c2094-istio-podinfo" pod "bfae7061-0326-46fd-901b-0399080c2094" (UID: "bfae7061-0326-46fd-901b-0399080c2094"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:08:02.496770 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:08:02.496714    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/025aea26-9742-4272-9249-2e916c03d53d-istio-podinfo" pod "025aea26-9742-4272-9249-2e916c03d53d" (UID: "025aea26-9742-4272-9249-2e916c03d53d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:08:02.497055 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:08:02.497030    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/025aea26-9742-4272-9249-2e916c03d53d-istio-token" pod "025aea26-9742-4272-9249-2e916c03d53d" (UID: "025aea26-9742-4272-9249-2e916c03d53d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:08:02.497355 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:08:02.497329    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfae7061-0326-46fd-901b-0399080c2094-kube-api-access-bxlzx" pod "bfae7061-0326-46fd-901b-0399080c2094" (UID: "bfae7061-0326-46fd-901b-0399080c2094"). InnerVolumeSpecName "kube-api-access-bxlzx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:08:02.497457 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:08:02.497414    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/025aea26-9742-4272-9249-2e916c03d53d-kube-api-access-45jmz" pod "025aea26-9742-4272-9249-2e916c03d53d" (UID: "025aea26-9742-4272-9249-2e916c03d53d"). InnerVolumeSpecName "kube-api-access-45jmz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:08:02.498065 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:08:02.498045    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfae7061-0326-46fd-901b-0399080c2094-istio-token" pod "bfae7061-0326-46fd-901b-0399080c2094" (UID: "bfae7061-0326-46fd-901b-0399080c2094"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:08:02.557574 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:08:02.557521    5061 generic.go:356] "Generic (PLEG): container finished" podID="bfae7061-0326-46fd-901b-0399080c2094" containerID="888f926dda2182b4bd0ce60084acaa9343c93fc6be5e86dc479441e7cc6f50be" exitCode=0
Oct 30 22:08:02.561591 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:08:02.558678    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-c79hw"
Oct 30 22:08:02.561591 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:08:02.558698    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-c79hw" event={"ID":"bfae7061-0326-46fd-901b-0399080c2094","Type":"ContainerDied","Data":"888f926dda2182b4bd0ce60084acaa9343c93fc6be5e86dc479441e7cc6f50be"}
Oct 30 22:08:02.561591 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:08:02.560990    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-c79hw" event={"ID":"bfae7061-0326-46fd-901b-0399080c2094","Type":"ContainerDied","Data":"5e84dbcf17016e163621b5fd14f1b7f9bc87e63c98173240ad51ba66466e0bc5"}
Oct 30 22:08:02.561591 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:08:02.561038    5061 scope.go:122] "RemoveContainer" containerID="888f926dda2182b4bd0ce60084acaa9343c93fc6be5e86dc479441e7cc6f50be"
Oct 30 22:08:02.564606 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:08:02.563585    5061 generic.go:356] "Generic (PLEG): container finished" podID="025aea26-9742-4272-9249-2e916c03d53d" containerID="20f371dff9c18ed9ac092238deb6a2806e4a79f27afa197072a9fa0f4aae236c" exitCode=0
Oct 30 22:08:02.564606 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:08:02.563665    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-wg5rk" event={"ID":"025aea26-9742-4272-9249-2e916c03d53d","Type":"ContainerDied","Data":"20f371dff9c18ed9ac092238deb6a2806e4a79f27afa197072a9fa0f4aae236c"}
Oct 30 22:08:02.564606 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:08:02.563698    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-wg5rk"
Oct 30 22:08:02.564606 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:08:02.563706    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-wg5rk" event={"ID":"025aea26-9742-4272-9249-2e916c03d53d","Type":"ContainerDied","Data":"6442fe84e95c817174a2792cdc26997d8658fdcd758283db8bdaeb2fb3baf61b"}
Oct 30 22:08:02.576793 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:08:02.576760    5061 scope.go:122] "RemoveContainer" containerID="888f926dda2182b4bd0ce60084acaa9343c93fc6be5e86dc479441e7cc6f50be"
Oct 30 22:08:02.577105 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:08:02.577082    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"888f926dda2182b4bd0ce60084acaa9343c93fc6be5e86dc479441e7cc6f50be\": container with ID starting with 888f926dda2182b4bd0ce60084acaa9343c93fc6be5e86dc479441e7cc6f50be not found: ID does not exist" containerID="888f926dda2182b4bd0ce60084acaa9343c93fc6be5e86dc479441e7cc6f50be"
Oct 30 22:08:02.577194 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:08:02.577116    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"888f926dda2182b4bd0ce60084acaa9343c93fc6be5e86dc479441e7cc6f50be"} err="failed to get container status \"888f926dda2182b4bd0ce60084acaa9343c93fc6be5e86dc479441e7cc6f50be\": rpc error: code = NotFound desc = could not find container \"888f926dda2182b4bd0ce60084acaa9343c93fc6be5e86dc479441e7cc6f50be\": container with ID starting with 888f926dda2182b4bd0ce60084acaa9343c93fc6be5e86dc479441e7cc6f50be not found: ID does not exist"
Oct 30 22:08:02.577194 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:08:02.577136    5061 scope.go:122] "RemoveContainer" containerID="20f371dff9c18ed9ac092238deb6a2806e4a79f27afa197072a9fa0f4aae236c"
Oct 30 22:08:02.589303 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:08:02.589271    5061 scope.go:122] "RemoveContainer" containerID="20f371dff9c18ed9ac092238deb6a2806e4a79f27afa197072a9fa0f4aae236c"
Oct 30 22:08:02.590605 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:08:02.589615    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20f371dff9c18ed9ac092238deb6a2806e4a79f27afa197072a9fa0f4aae236c\": container with ID starting with 20f371dff9c18ed9ac092238deb6a2806e4a79f27afa197072a9fa0f4aae236c not found: ID does not exist" containerID="20f371dff9c18ed9ac092238deb6a2806e4a79f27afa197072a9fa0f4aae236c"
Oct 30 22:08:02.590605 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:08:02.589648    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20f371dff9c18ed9ac092238deb6a2806e4a79f27afa197072a9fa0f4aae236c"} err="failed to get container status \"20f371dff9c18ed9ac092238deb6a2806e4a79f27afa197072a9fa0f4aae236c\": rpc error: code = NotFound desc = could not find container \"20f371dff9c18ed9ac092238deb6a2806e4a79f27afa197072a9fa0f4aae236c\": container with ID starting with 20f371dff9c18ed9ac092238deb6a2806e4a79f27afa197072a9fa0f4aae236c not found: ID does not exist"
Oct 30 22:08:02.597435 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:08:02.597408    5061 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/025aea26-9742-4272-9249-2e916c03d53d-credential-socket\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:02.597611 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:08:02.597590    5061 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/025aea26-9742-4272-9249-2e916c03d53d-istio-podinfo\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:02.597721 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:08:02.597614    5061 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/025aea26-9742-4272-9249-2e916c03d53d-istio-envoy\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:02.597721 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:08:02.597625    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxlzx\" (UniqueName: \"kubernetes.io/projected/bfae7061-0326-46fd-901b-0399080c2094-kube-api-access-bxlzx\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:02.597721 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:08:02.597635    5061 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfae7061-0326-46fd-901b-0399080c2094-istio-envoy\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:02.597721 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:08:02.597644    5061 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfae7061-0326-46fd-901b-0399080c2094-istio-podinfo\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:02.597721 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:08:02.597653    5061 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/025aea26-9742-4272-9249-2e916c03d53d-istio-token\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:02.597721 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:08:02.597662    5061 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfae7061-0326-46fd-901b-0399080c2094-istio-token\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:02.597721 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:08:02.597670    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45jmz\" (UniqueName: \"kubernetes.io/projected/025aea26-9742-4272-9249-2e916c03d53d-kube-api-access-45jmz\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:02.800442 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:08:02.800404    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-c79hw"]
Oct 30 22:08:02.814454 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:08:02.814383    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-c79hw"]
Oct 30 22:08:02.927775 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:08:02.927738    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-wg5rk"]
Oct 30 22:08:02.932557 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:08:02.932502    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-wg5rk"]
Oct 30 22:08:04.530291 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:08:04.530257    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="025aea26-9742-4272-9249-2e916c03d53d" path="/var/lib/kubelet/pods/025aea26-9742-4272-9249-2e916c03d53d/volumes"
Oct 30 22:08:04.530985 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:08:04.530964    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfae7061-0326-46fd-901b-0399080c2094" path="/var/lib/kubelet/pods/bfae7061-0326-46fd-901b-0399080c2094/volumes"
Oct 30 22:08:24.669727 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:08:24.669683    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 22:08:28.024177 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:08:28.024141    5061 scope.go:122] "RemoveContainer" containerID="42ef4afa9fddcbc85c0000a3c7f2e3a496f040fae73f285cdfc10a484e47386c"
Oct 30 22:08:53.095626 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:08:53.095518    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8123afac-hn7mk"]
Oct 30 22:08:53.146286 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:08:53.146248    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8123afac-hn7mk"]
Oct 30 22:08:54.486804 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:08:54.486770    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-7d434591-openshift-default-58f74c4865-rm5js"]
Oct 30 22:08:54.487803 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:08:54.487780    5061 cpu_manager.go:414] "containerMap: removing container" podUID="025aea26-9742-4272-9249-2e916c03d53d" containerName="istio-proxy"
Oct 30 22:08:54.487977 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:08:54.487957    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="025aea26-9742-4272-9249-2e916c03d53d" containerName="istio-proxy"
Oct 30 22:08:54.488123 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:08:54.487983    5061 cpu_manager.go:414] "containerMap: removing container" podUID="bfae7061-0326-46fd-901b-0399080c2094" containerName="istio-proxy"
Oct 30 22:08:54.488123 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:08:54.487992    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfae7061-0326-46fd-901b-0399080c2094" containerName="istio-proxy"
Oct 30 22:08:54.488123 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:08:54.488090    5061 cpu_manager.go:414] "containerMap: removing container" podUID="bd1c8a85-674c-4888-8f28-de388047d1ca" containerName="istio-proxy"
Oct 30 22:08:54.488123 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:08:54.488101    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd1c8a85-674c-4888-8f28-de388047d1ca" containerName="istio-proxy"
Oct 30 22:08:54.488331 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:08:54.488130    5061 cpu_manager.go:414] "containerMap: removing container" podUID="33935040-0f5f-43db-91c7-cae4d0e37369" containerName="istio-proxy"
Oct 30 22:08:54.488331 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:08:54.488137    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33935040-0f5f-43db-91c7-cae4d0e37369" containerName="istio-proxy"
Oct 30 22:08:54.488432 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:08:54.488359    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd1c8a85-674c-4888-8f28-de388047d1ca" containerName="istio-proxy"
Oct 30 22:08:54.488432 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:08:54.488377    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="33935040-0f5f-43db-91c7-cae4d0e37369" containerName="istio-proxy"
Oct 30 22:08:54.488432 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:08:54.488391    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="025aea26-9742-4272-9249-2e916c03d53d" containerName="istio-proxy"
Oct 30 22:08:54.488432 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:08:54.488400    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfae7061-0326-46fd-901b-0399080c2094" containerName="istio-proxy"
Oct 30 22:08:54.493932 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:08:54.493910    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7d434591-openshift-default-58f74c4865-rm5js"
Oct 30 22:08:54.537453 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:08:54.537418    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79096c68-4183-43fc-8262-2668e2f26057" path="/var/lib/kubelet/pods/79096c68-4183-43fc-8262-2668e2f26057/volumes"
Oct 30 22:08:54.590242 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:08:54.590191    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-7d434591-openshift-default-dockercfg-pndqd\""
Oct 30 22:08:54.615009 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:08:54.610700    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce46d672-35a3-4cca-a9fc-b070939de95f-istio-podinfo\") pod \"e2e-mt-7d434591-openshift-default-58f74c4865-rm5js\" (UID: \"ce46d672-35a3-4cca-a9fc-b070939de95f\") " pod="openshift-ingress/e2e-mt-7d434591-openshift-default-58f74c4865-rm5js"
Oct 30 22:08:54.615293 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:08:54.615268    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce46d672-35a3-4cca-a9fc-b070939de95f-workload-certs\") pod \"e2e-mt-7d434591-openshift-default-58f74c4865-rm5js\" (UID: \"ce46d672-35a3-4cca-a9fc-b070939de95f\") " pod="openshift-ingress/e2e-mt-7d434591-openshift-default-58f74c4865-rm5js"
Oct 30 22:08:54.615585 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:08:54.615558    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce46d672-35a3-4cca-a9fc-b070939de95f-credential-socket\") pod \"e2e-mt-7d434591-openshift-default-58f74c4865-rm5js\" (UID: \"ce46d672-35a3-4cca-a9fc-b070939de95f\") " pod="openshift-ingress/e2e-mt-7d434591-openshift-default-58f74c4865-rm5js"
Oct 30 22:08:54.615749 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:08:54.615722    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce46d672-35a3-4cca-a9fc-b070939de95f-workload-socket\") pod \"e2e-mt-7d434591-openshift-default-58f74c4865-rm5js\" (UID: \"ce46d672-35a3-4cca-a9fc-b070939de95f\") " pod="openshift-ingress/e2e-mt-7d434591-openshift-default-58f74c4865-rm5js"
Oct 30 22:08:54.615877 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:08:54.615861    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce46d672-35a3-4cca-a9fc-b070939de95f-istio-data\") pod \"e2e-mt-7d434591-openshift-default-58f74c4865-rm5js\" (UID: \"ce46d672-35a3-4cca-a9fc-b070939de95f\") " pod="openshift-ingress/e2e-mt-7d434591-openshift-default-58f74c4865-rm5js"
Oct 30 22:08:54.615959 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:08:54.615946    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce46d672-35a3-4cca-a9fc-b070939de95f-istio-envoy\") pod \"e2e-mt-7d434591-openshift-default-58f74c4865-rm5js\" (UID: \"ce46d672-35a3-4cca-a9fc-b070939de95f\") " pod="openshift-ingress/e2e-mt-7d434591-openshift-default-58f74c4865-rm5js"
Oct 30 22:08:54.616016 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:08:54.615989    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce46d672-35a3-4cca-a9fc-b070939de95f-istiod-ca-cert\") pod \"e2e-mt-7d434591-openshift-default-58f74c4865-rm5js\" (UID: \"ce46d672-35a3-4cca-a9fc-b070939de95f\") " pod="openshift-ingress/e2e-mt-7d434591-openshift-default-58f74c4865-rm5js"
Oct 30 22:08:54.616121 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:08:54.616109    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce46d672-35a3-4cca-a9fc-b070939de95f-istio-token\") pod \"e2e-mt-7d434591-openshift-default-58f74c4865-rm5js\" (UID: \"ce46d672-35a3-4cca-a9fc-b070939de95f\") " pod="openshift-ingress/e2e-mt-7d434591-openshift-default-58f74c4865-rm5js"
Oct 30 22:08:54.616229 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:08:54.616214    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgh9p\" (UniqueName: \"kubernetes.io/projected/ce46d672-35a3-4cca-a9fc-b070939de95f-kube-api-access-fgh9p\") pod \"e2e-mt-7d434591-openshift-default-58f74c4865-rm5js\" (UID: \"ce46d672-35a3-4cca-a9fc-b070939de95f\") " pod="openshift-ingress/e2e-mt-7d434591-openshift-default-58f74c4865-rm5js"
Oct 30 22:08:54.663149 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:08:54.663123    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-7d434591-openshift-default-58f74c4865-rm5js"]
Oct 30 22:08:54.710910 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:08:54.710261    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 22:08:54.721624 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:08:54.720540    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce46d672-35a3-4cca-a9fc-b070939de95f-istio-podinfo\") pod \"e2e-mt-7d434591-openshift-default-58f74c4865-rm5js\" (UID: \"ce46d672-35a3-4cca-a9fc-b070939de95f\") " pod="openshift-ingress/e2e-mt-7d434591-openshift-default-58f74c4865-rm5js"
Oct 30 22:08:54.721624 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:08:54.720589    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce46d672-35a3-4cca-a9fc-b070939de95f-workload-certs\") pod \"e2e-mt-7d434591-openshift-default-58f74c4865-rm5js\" (UID: \"ce46d672-35a3-4cca-a9fc-b070939de95f\") " pod="openshift-ingress/e2e-mt-7d434591-openshift-default-58f74c4865-rm5js"
Oct 30 22:08:54.721624 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:08:54.720652    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce46d672-35a3-4cca-a9fc-b070939de95f-credential-socket\") pod \"e2e-mt-7d434591-openshift-default-58f74c4865-rm5js\" (UID: \"ce46d672-35a3-4cca-a9fc-b070939de95f\") " pod="openshift-ingress/e2e-mt-7d434591-openshift-default-58f74c4865-rm5js"
Oct 30 22:08:54.721624 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:08:54.720686    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce46d672-35a3-4cca-a9fc-b070939de95f-workload-socket\") pod \"e2e-mt-7d434591-openshift-default-58f74c4865-rm5js\" (UID: \"ce46d672-35a3-4cca-a9fc-b070939de95f\") " pod="openshift-ingress/e2e-mt-7d434591-openshift-default-58f74c4865-rm5js"
Oct 30 22:08:54.721624 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:08:54.720784    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce46d672-35a3-4cca-a9fc-b070939de95f-istio-data\") pod \"e2e-mt-7d434591-openshift-default-58f74c4865-rm5js\" (UID: \"ce46d672-35a3-4cca-a9fc-b070939de95f\") " pod="openshift-ingress/e2e-mt-7d434591-openshift-default-58f74c4865-rm5js"
Oct 30 22:08:54.721624 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:08:54.720854    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce46d672-35a3-4cca-a9fc-b070939de95f-istio-envoy\") pod \"e2e-mt-7d434591-openshift-default-58f74c4865-rm5js\" (UID: \"ce46d672-35a3-4cca-a9fc-b070939de95f\") " pod="openshift-ingress/e2e-mt-7d434591-openshift-default-58f74c4865-rm5js"
Oct 30 22:08:54.721624 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:08:54.720890    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce46d672-35a3-4cca-a9fc-b070939de95f-istiod-ca-cert\") pod \"e2e-mt-7d434591-openshift-default-58f74c4865-rm5js\" (UID: \"ce46d672-35a3-4cca-a9fc-b070939de95f\") " pod="openshift-ingress/e2e-mt-7d434591-openshift-default-58f74c4865-rm5js"
Oct 30 22:08:54.721624 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:08:54.720996    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce46d672-35a3-4cca-a9fc-b070939de95f-istio-token\") pod \"e2e-mt-7d434591-openshift-default-58f74c4865-rm5js\" (UID: \"ce46d672-35a3-4cca-a9fc-b070939de95f\") " pod="openshift-ingress/e2e-mt-7d434591-openshift-default-58f74c4865-rm5js"
Oct 30 22:08:54.721624 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:08:54.721090    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgh9p\" (UniqueName: \"kubernetes.io/projected/ce46d672-35a3-4cca-a9fc-b070939de95f-kube-api-access-fgh9p\") pod \"e2e-mt-7d434591-openshift-default-58f74c4865-rm5js\" (UID: \"ce46d672-35a3-4cca-a9fc-b070939de95f\") " pod="openshift-ingress/e2e-mt-7d434591-openshift-default-58f74c4865-rm5js"
Oct 30 22:08:54.722184 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:08:54.721879    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce46d672-35a3-4cca-a9fc-b070939de95f-istio-data\") pod \"e2e-mt-7d434591-openshift-default-58f74c4865-rm5js\" (UID: \"ce46d672-35a3-4cca-a9fc-b070939de95f\") " pod="openshift-ingress/e2e-mt-7d434591-openshift-default-58f74c4865-rm5js"
Oct 30 22:08:54.722184 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:08:54.722143    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce46d672-35a3-4cca-a9fc-b070939de95f-workload-certs\") pod \"e2e-mt-7d434591-openshift-default-58f74c4865-rm5js\" (UID: \"ce46d672-35a3-4cca-a9fc-b070939de95f\") " pod="openshift-ingress/e2e-mt-7d434591-openshift-default-58f74c4865-rm5js"
Oct 30 22:08:54.722803 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:08:54.722392    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce46d672-35a3-4cca-a9fc-b070939de95f-credential-socket\") pod \"e2e-mt-7d434591-openshift-default-58f74c4865-rm5js\" (UID: \"ce46d672-35a3-4cca-a9fc-b070939de95f\") " pod="openshift-ingress/e2e-mt-7d434591-openshift-default-58f74c4865-rm5js"
Oct 30 22:08:54.723302 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:08:54.723261    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce46d672-35a3-4cca-a9fc-b070939de95f-workload-socket\") pod \"e2e-mt-7d434591-openshift-default-58f74c4865-rm5js\" (UID: \"ce46d672-35a3-4cca-a9fc-b070939de95f\") " pod="openshift-ingress/e2e-mt-7d434591-openshift-default-58f74c4865-rm5js"
Oct 30 22:08:54.723394 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:08:54.723341    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce46d672-35a3-4cca-a9fc-b070939de95f-istiod-ca-cert\") pod \"e2e-mt-7d434591-openshift-default-58f74c4865-rm5js\" (UID: \"ce46d672-35a3-4cca-a9fc-b070939de95f\") " pod="openshift-ingress/e2e-mt-7d434591-openshift-default-58f74c4865-rm5js"
Oct 30 22:08:54.726745 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:08:54.726720    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce46d672-35a3-4cca-a9fc-b070939de95f-istio-podinfo\") pod \"e2e-mt-7d434591-openshift-default-58f74c4865-rm5js\" (UID: \"ce46d672-35a3-4cca-a9fc-b070939de95f\") " pod="openshift-ingress/e2e-mt-7d434591-openshift-default-58f74c4865-rm5js"
Oct 30 22:08:54.730185 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:08:54.730162    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce46d672-35a3-4cca-a9fc-b070939de95f-istio-envoy\") pod \"e2e-mt-7d434591-openshift-default-58f74c4865-rm5js\" (UID: \"ce46d672-35a3-4cca-a9fc-b070939de95f\") " pod="openshift-ingress/e2e-mt-7d434591-openshift-default-58f74c4865-rm5js"
Oct 30 22:08:54.805719 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:08:54.805680    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce46d672-35a3-4cca-a9fc-b070939de95f-istio-token\") pod \"e2e-mt-7d434591-openshift-default-58f74c4865-rm5js\" (UID: \"ce46d672-35a3-4cca-a9fc-b070939de95f\") " pod="openshift-ingress/e2e-mt-7d434591-openshift-default-58f74c4865-rm5js"
Oct 30 22:08:54.805876 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:08:54.805829    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgh9p\" (UniqueName: \"kubernetes.io/projected/ce46d672-35a3-4cca-a9fc-b070939de95f-kube-api-access-fgh9p\") pod \"e2e-mt-7d434591-openshift-default-58f74c4865-rm5js\" (UID: \"ce46d672-35a3-4cca-a9fc-b070939de95f\") " pod="openshift-ingress/e2e-mt-7d434591-openshift-default-58f74c4865-rm5js"
Oct 30 22:08:54.823162 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:08:54.823112    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7d434591-openshift-default-58f74c4865-rm5js"
Oct 30 22:08:55.290266 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:08:55.290183    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-7d434591-openshift-default-58f74c4865-rm5js"]
Oct 30 22:08:55.307073 ip-10-0-130-60 kubenswrapper[5061]: W1030 22:08:55.307041    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce46d672_35a3_4cca_a9fc_b070939de95f.slice/crio-a750f773a54be887ded3179581f4bd3b96f4734168de0c23f099e67bcf841fe3 WatchSource:0}: Error finding container a750f773a54be887ded3179581f4bd3b96f4734168de0c23f099e67bcf841fe3: Status 404 returned error can't find the container with id a750f773a54be887ded3179581f4bd3b96f4734168de0c23f099e67bcf841fe3
Oct 30 22:08:55.309455 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:08:55.309424    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:08:55.309546 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:08:55.309492    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:08:55.309546 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:08:55.309521    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:08:56.067443 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:08:56.067407    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7d434591-openshift-default-58f74c4865-rm5js" event={"ID":"ce46d672-35a3-4cca-a9fc-b070939de95f","Type":"ContainerStarted","Data":"614f26a23cbceea26931c4ad079465625aa89f7b37390644c6d0999bb51b7970"}
Oct 30 22:08:56.067968 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:08:56.067470    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7d434591-openshift-default-58f74c4865-rm5js" event={"ID":"ce46d672-35a3-4cca-a9fc-b070939de95f","Type":"ContainerStarted","Data":"a750f773a54be887ded3179581f4bd3b96f4734168de0c23f099e67bcf841fe3"}
Oct 30 22:08:56.824313 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:08:56.824280    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-7d434591-openshift-default-58f74c4865-rm5js"
Oct 30 22:08:56.830084 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:08:56.830058    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-7d434591-openshift-default-58f74c4865-rm5js"
Oct 30 22:08:57.019439 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:08:57.019383    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-7d434591-openshift-default-58f74c4865-rm5js" podStartSLOduration=3.019369453 podStartE2EDuration="3.019369453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:08:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:08:56.24992767 +0000 UTC m=+1589.862697446" watchObservedRunningTime="2026-10-30 22:08:57.019369453 +0000 UTC m=+1590.632139211"
Oct 30 22:08:57.076202 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:08:57.076125    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-7d434591-openshift-default-58f74c4865-rm5js"
Oct 30 22:08:57.077202 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:08:57.077179    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-7d434591-openshift-default-58f74c4865-rm5js"
Oct 30 22:09:05.064572 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:09:05.064509    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27350177-mdzkw"]
Oct 30 22:09:05.117347 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:09:05.117310    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27350177-mdzkw"]
Oct 30 22:09:06.065805 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:09:06.065767    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8c7d3fc-qjfvv"]
Oct 30 22:09:06.121004 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:09:06.120968    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8c7d3fc-qjfvv"]
Oct 30 22:09:06.530769 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:09:06.530695    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7780bf6d-2d91-4d60-b073-6cc0b1b7009e" path="/var/lib/kubelet/pods/7780bf6d-2d91-4d60-b073-6cc0b1b7009e/volumes"
Oct 30 22:09:06.531184 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:09:06.531170    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9665869-1cbf-484c-89e1-d25f21b7ec54" path="/var/lib/kubelet/pods/c9665869-1cbf-484c-89e1-d25f21b7ec54/volumes"
Oct 30 22:09:15.327345 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:09:15.327313    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-13297892-66644cbc47-k8xjl"]
Oct 30 22:09:20.328629 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:09:20.328589    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-13297892-66644cbc47-k8xjl" podUID="a339a53c-1e2e-45e3-af4a-1538cc2b7e00" containerName="payload-pre-processing" containerID="cri-o://a22ae8c774b95b466271922f96492e4ac0efdc143c5f480a17de7f68fc620190" gracePeriod=25
Oct 30 22:09:20.630770 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:09:20.630738    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-13297892-66644cbc47-k8xjl"
Oct 30 22:09:20.711427 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:09:20.711395    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a339a53c-1e2e-45e3-af4a-1538cc2b7e00-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a339a53c-1e2e-45e3-af4a-1538cc2b7e00-plugins-config-volume\") pod \"a339a53c-1e2e-45e3-af4a-1538cc2b7e00\" (UID: \"a339a53c-1e2e-45e3-af4a-1538cc2b7e00\") "
Oct 30 22:09:20.711656 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:09:20.711447    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a339a53c-1e2e-45e3-af4a-1538cc2b7e00-kube-api-access-9k2wb\" (UniqueName: \"kubernetes.io/projected/a339a53c-1e2e-45e3-af4a-1538cc2b7e00-kube-api-access-9k2wb\") pod \"a339a53c-1e2e-45e3-af4a-1538cc2b7e00\" (UID: \"a339a53c-1e2e-45e3-af4a-1538cc2b7e00\") "
Oct 30 22:09:20.711800 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:09:20.711778    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a339a53c-1e2e-45e3-af4a-1538cc2b7e00-plugins-config-volume" pod "a339a53c-1e2e-45e3-af4a-1538cc2b7e00" (UID: "a339a53c-1e2e-45e3-af4a-1538cc2b7e00"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:09:20.712137 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:09:20.712115    5061 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a339a53c-1e2e-45e3-af4a-1538cc2b7e00-plugins-config-volume\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:09:20.713719 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:09:20.713690    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a339a53c-1e2e-45e3-af4a-1538cc2b7e00-kube-api-access-9k2wb" pod "a339a53c-1e2e-45e3-af4a-1538cc2b7e00" (UID: "a339a53c-1e2e-45e3-af4a-1538cc2b7e00"). InnerVolumeSpecName "kube-api-access-9k2wb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:09:20.814873 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:09:20.814832    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9k2wb\" (UniqueName: \"kubernetes.io/projected/a339a53c-1e2e-45e3-af4a-1538cc2b7e00-kube-api-access-9k2wb\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:09:21.304358 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:09:21.304326    5061 generic.go:356] "Generic (PLEG): container finished" podID="a339a53c-1e2e-45e3-af4a-1538cc2b7e00" containerID="a22ae8c774b95b466271922f96492e4ac0efdc143c5f480a17de7f68fc620190" exitCode=0
Oct 30 22:09:21.304589 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:09:21.304370    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-13297892-66644cbc47-k8xjl"
Oct 30 22:09:21.304589 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:09:21.304384    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-13297892-66644cbc47-k8xjl" event={"ID":"a339a53c-1e2e-45e3-af4a-1538cc2b7e00","Type":"ContainerDied","Data":"a22ae8c774b95b466271922f96492e4ac0efdc143c5f480a17de7f68fc620190"}
Oct 30 22:09:21.304589 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:09:21.304429    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-13297892-66644cbc47-k8xjl" event={"ID":"a339a53c-1e2e-45e3-af4a-1538cc2b7e00","Type":"ContainerDied","Data":"645d35f5da50185616427c1768b0bb86ee5c2bcd9bc8f4eb331923f7862715ee"}
Oct 30 22:09:21.304589 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:09:21.304450    5061 scope.go:122] "RemoveContainer" containerID="a22ae8c774b95b466271922f96492e4ac0efdc143c5f480a17de7f68fc620190"
Oct 30 22:09:21.315386 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:09:21.315367    5061 scope.go:122] "RemoveContainer" containerID="a22ae8c774b95b466271922f96492e4ac0efdc143c5f480a17de7f68fc620190"
Oct 30 22:09:21.315646 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:09:21.315624    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a22ae8c774b95b466271922f96492e4ac0efdc143c5f480a17de7f68fc620190\": container with ID starting with a22ae8c774b95b466271922f96492e4ac0efdc143c5f480a17de7f68fc620190 not found: ID does not exist" containerID="a22ae8c774b95b466271922f96492e4ac0efdc143c5f480a17de7f68fc620190"
Oct 30 22:09:21.315716 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:09:21.315657    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a22ae8c774b95b466271922f96492e4ac0efdc143c5f480a17de7f68fc620190"} err="failed to get container status \"a22ae8c774b95b466271922f96492e4ac0efdc143c5f480a17de7f68fc620190\": rpc error: code = NotFound desc = could not find container \"a22ae8c774b95b466271922f96492e4ac0efdc143c5f480a17de7f68fc620190\": container with ID starting with a22ae8c774b95b466271922f96492e4ac0efdc143c5f480a17de7f68fc620190 not found: ID does not exist"
Oct 30 22:09:21.484879 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:09:21.484846    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-13297892-66644cbc47-k8xjl"]
Oct 30 22:09:21.537319 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:09:21.537282    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-13297892-66644cbc47-k8xjl"]
Oct 30 22:09:22.053883 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:09:22.053848    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42e4d1eb-58l8b"]
Oct 30 22:09:22.057566 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:09:22.057539    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42e4d1eb-58l8b"]
Oct 30 22:09:22.537175 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:09:22.537060    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a339a53c-1e2e-45e3-af4a-1538cc2b7e00" path="/var/lib/kubelet/pods/a339a53c-1e2e-45e3-af4a-1538cc2b7e00/volumes"
Oct 30 22:09:22.537701 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:09:22.537684    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5fa1ec3-cd5d-4837-a869-e94c66c445cd" path="/var/lib/kubelet/pods/d5fa1ec3-cd5d-4837-a869-e94c66c445cd/volumes"
Oct 30 22:09:22.611371 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:09:22.611334    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7d434591-6f89ff7d59-kbg9t"]
Oct 30 22:09:22.612547 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:09:22.612504    5061 cpu_manager.go:414] "containerMap: removing container" podUID="a339a53c-1e2e-45e3-af4a-1538cc2b7e00" containerName="payload-pre-processing"
Oct 30 22:09:22.612547 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:09:22.612544    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a339a53c-1e2e-45e3-af4a-1538cc2b7e00" containerName="payload-pre-processing"
Oct 30 22:09:22.612773 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:09:22.612757    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="a339a53c-1e2e-45e3-af4a-1538cc2b7e00" containerName="payload-pre-processing"
Oct 30 22:09:22.618315 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:09:22.618289    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7d434591-6f89ff7d59-kbg9t"
Oct 30 22:09:22.678182 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:09:22.678136    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7d434591-6f89ff7d59-kbg9t"]
Oct 30 22:09:22.727792 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:09:22.727761    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-7d434591\""
Oct 30 22:09:22.727970 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:09:22.727882    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-7d434591-dockercfg-ppccn\""
Oct 30 22:09:22.746919 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:09:22.746879    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkmw7\" (UniqueName: \"kubernetes.io/projected/6e43c8db-84a2-453e-8bac-04f149ddd7a6-kube-api-access-kkmw7\") pod \"payload-pre-processing-e2e-mt-7d434591-6f89ff7d59-kbg9t\" (UID: \"6e43c8db-84a2-453e-8bac-04f149ddd7a6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7d434591-6f89ff7d59-kbg9t"
Oct 30 22:09:22.747091 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:09:22.747044    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e43c8db-84a2-453e-8bac-04f149ddd7a6-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-7d434591-6f89ff7d59-kbg9t\" (UID: \"6e43c8db-84a2-453e-8bac-04f149ddd7a6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7d434591-6f89ff7d59-kbg9t"
Oct 30 22:09:22.848596 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:09:22.848558    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkmw7\" (UniqueName: \"kubernetes.io/projected/6e43c8db-84a2-453e-8bac-04f149ddd7a6-kube-api-access-kkmw7\") pod \"payload-pre-processing-e2e-mt-7d434591-6f89ff7d59-kbg9t\" (UID: \"6e43c8db-84a2-453e-8bac-04f149ddd7a6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7d434591-6f89ff7d59-kbg9t"
Oct 30 22:09:22.848776 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:09:22.848750    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e43c8db-84a2-453e-8bac-04f149ddd7a6-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-7d434591-6f89ff7d59-kbg9t\" (UID: \"6e43c8db-84a2-453e-8bac-04f149ddd7a6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7d434591-6f89ff7d59-kbg9t"
Oct 30 22:09:22.849463 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:09:22.849445    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e43c8db-84a2-453e-8bac-04f149ddd7a6-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-7d434591-6f89ff7d59-kbg9t\" (UID: \"6e43c8db-84a2-453e-8bac-04f149ddd7a6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7d434591-6f89ff7d59-kbg9t"
Oct 30 22:09:22.909909 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:09:22.909877    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkmw7\" (UniqueName: \"kubernetes.io/projected/6e43c8db-84a2-453e-8bac-04f149ddd7a6-kube-api-access-kkmw7\") pod \"payload-pre-processing-e2e-mt-7d434591-6f89ff7d59-kbg9t\" (UID: \"6e43c8db-84a2-453e-8bac-04f149ddd7a6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7d434591-6f89ff7d59-kbg9t"
Oct 30 22:09:22.934718 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:09:22.934681    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7d434591-6f89ff7d59-kbg9t"
Oct 30 22:09:23.360771 ip-10-0-130-60 kubenswrapper[5061]: W1030 22:09:23.360733    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e43c8db_84a2_453e_8bac_04f149ddd7a6.slice/crio-fc60c58c87f483cd089540ec8cd6a8ee2038bbe748d13f8ab26e851d52dccfd1 WatchSource:0}: Error finding container fc60c58c87f483cd089540ec8cd6a8ee2038bbe748d13f8ab26e851d52dccfd1: Status 404 returned error can't find the container with id fc60c58c87f483cd089540ec8cd6a8ee2038bbe748d13f8ab26e851d52dccfd1
Oct 30 22:09:23.366629 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:09:23.366608    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7d434591-6f89ff7d59-kbg9t"]
Oct 30 22:09:24.352018 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:09:24.351974    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7d434591-6f89ff7d59-kbg9t" event={"ID":"6e43c8db-84a2-453e-8bac-04f149ddd7a6","Type":"ContainerStarted","Data":"b5aee796a5c954dbed27815703e412db7c6d3e0911e5b53536e663192732a29f"}
Oct 30 22:09:24.352749 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:09:24.352073    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7d434591-6f89ff7d59-kbg9t" event={"ID":"6e43c8db-84a2-453e-8bac-04f149ddd7a6","Type":"ContainerStarted","Data":"fc60c58c87f483cd089540ec8cd6a8ee2038bbe748d13f8ab26e851d52dccfd1"}
Oct 30 22:09:24.352749 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:09:24.352246    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-7d434591-6f89ff7d59-kbg9t"
Oct 30 22:09:24.473558 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:09:24.473494    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-7d434591-6f89ff7d59-kbg9t" podStartSLOduration=2.473482133 podStartE2EDuration="2.473482133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:09:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:09:24.471134097 +0000 UTC m=+1618.083903855" watchObservedRunningTime="2026-10-30 22:09:24.473482133 +0000 UTC m=+1618.086251890"
Oct 30 22:09:24.676158 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:09:24.676068    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 22:09:25.053042 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:09:25.053011    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-998abd03-txgxm"]
Oct 30 22:09:25.056967 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:09:25.056939    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-998abd03-txgxm"]
Oct 30 22:09:26.531072 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:09:26.531042    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cad0f17c-1e87-406f-848d-d01ea9ba046f" path="/var/lib/kubelet/pods/cad0f17c-1e87-406f-848d-d01ea9ba046f/volumes"
Oct 30 22:09:35.059424 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:09:35.059390    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51bdef55-pvst4"]
Oct 30 22:09:35.112378 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:09:35.112342    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51bdef55-pvst4"]
Oct 30 22:09:35.396173 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:09:35.396096    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-7d434591-6f89ff7d59-kbg9t"
Oct 30 22:09:36.531067 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:09:36.531035    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7c1a2ce-12a9-4e6a-8586-5d39f62059fa" path="/var/lib/kubelet/pods/c7c1a2ce-12a9-4e6a-8586-5d39f62059fa/volumes"
Oct 30 22:09:53.052952 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:09:53.052918    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37cbad61-v9npg"]
Oct 30 22:09:53.104805 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:09:53.104776    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37cbad61-v9npg"]
Oct 30 22:09:54.531880 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:09:54.531843    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d82b81f-d252-4fd6-ac51-bc4dccbce83f" path="/var/lib/kubelet/pods/1d82b81f-d252-4fd6-ac51-bc4dccbce83f/volumes"
Oct 30 22:09:54.673427 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:09:54.673386    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 22:10:00.882618 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:10:00.882579    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-9r5c8"]
Oct 30 22:10:00.888333 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:10:00.888313    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-9r5c8"
Oct 30 22:10:00.948337 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:10:00.948305    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-778d39d3-openshift-default-dockercfg-9jkcd\""
Oct 30 22:10:00.983343 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:10:00.983304    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83adee7e-3585-4d0d-b1b2-d4b5d3be0588-istio-data\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-9r5c8\" (UID: \"83adee7e-3585-4d0d-b1b2-d4b5d3be0588\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-9r5c8"
Oct 30 22:10:00.983635 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:10:00.983612    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83adee7e-3585-4d0d-b1b2-d4b5d3be0588-istiod-ca-cert\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-9r5c8\" (UID: \"83adee7e-3585-4d0d-b1b2-d4b5d3be0588\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-9r5c8"
Oct 30 22:10:00.983944 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:10:00.983919    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83adee7e-3585-4d0d-b1b2-d4b5d3be0588-istio-token\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-9r5c8\" (UID: \"83adee7e-3585-4d0d-b1b2-d4b5d3be0588\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-9r5c8"
Oct 30 22:10:00.984043 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:10:00.983969    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83adee7e-3585-4d0d-b1b2-d4b5d3be0588-istio-envoy\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-9r5c8\" (UID: \"83adee7e-3585-4d0d-b1b2-d4b5d3be0588\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-9r5c8"
Oct 30 22:10:00.984043 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:10:00.984017    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83adee7e-3585-4d0d-b1b2-d4b5d3be0588-istio-podinfo\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-9r5c8\" (UID: \"83adee7e-3585-4d0d-b1b2-d4b5d3be0588\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-9r5c8"
Oct 30 22:10:00.984174 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:10:00.984051    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83adee7e-3585-4d0d-b1b2-d4b5d3be0588-workload-socket\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-9r5c8\" (UID: \"83adee7e-3585-4d0d-b1b2-d4b5d3be0588\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-9r5c8"
Oct 30 22:10:00.984174 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:10:00.984156    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83adee7e-3585-4d0d-b1b2-d4b5d3be0588-credential-socket\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-9r5c8\" (UID: \"83adee7e-3585-4d0d-b1b2-d4b5d3be0588\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-9r5c8"
Oct 30 22:10:00.984288 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:10:00.984227    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83adee7e-3585-4d0d-b1b2-d4b5d3be0588-workload-certs\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-9r5c8\" (UID: \"83adee7e-3585-4d0d-b1b2-d4b5d3be0588\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-9r5c8"
Oct 30 22:10:00.984288 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:10:00.984259    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqs79\" (UniqueName: \"kubernetes.io/projected/83adee7e-3585-4d0d-b1b2-d4b5d3be0588-kube-api-access-gqs79\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-9r5c8\" (UID: \"83adee7e-3585-4d0d-b1b2-d4b5d3be0588\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-9r5c8"
Oct 30 22:10:01.076236 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:10:01.076199    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-9r5c8"]
Oct 30 22:10:01.093877 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:10:01.093829    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83adee7e-3585-4d0d-b1b2-d4b5d3be0588-workload-socket\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-9r5c8\" (UID: \"83adee7e-3585-4d0d-b1b2-d4b5d3be0588\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-9r5c8"
Oct 30 22:10:01.096827 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:10:01.096777    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83adee7e-3585-4d0d-b1b2-d4b5d3be0588-credential-socket\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-9r5c8\" (UID: \"83adee7e-3585-4d0d-b1b2-d4b5d3be0588\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-9r5c8"
Oct 30 22:10:01.097371 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:10:01.097346    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83adee7e-3585-4d0d-b1b2-d4b5d3be0588-workload-certs\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-9r5c8\" (UID: \"83adee7e-3585-4d0d-b1b2-d4b5d3be0588\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-9r5c8"
Oct 30 22:10:01.097719 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:10:01.097698    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqs79\" (UniqueName: \"kubernetes.io/projected/83adee7e-3585-4d0d-b1b2-d4b5d3be0588-kube-api-access-gqs79\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-9r5c8\" (UID: \"83adee7e-3585-4d0d-b1b2-d4b5d3be0588\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-9r5c8"
Oct 30 22:10:01.099747 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:10:01.099723    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83adee7e-3585-4d0d-b1b2-d4b5d3be0588-istio-data\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-9r5c8\" (UID: \"83adee7e-3585-4d0d-b1b2-d4b5d3be0588\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-9r5c8"
Oct 30 22:10:01.102163 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:10:01.102135    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83adee7e-3585-4d0d-b1b2-d4b5d3be0588-istiod-ca-cert\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-9r5c8\" (UID: \"83adee7e-3585-4d0d-b1b2-d4b5d3be0588\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-9r5c8"
Oct 30 22:10:01.102493 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:10:01.102448    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83adee7e-3585-4d0d-b1b2-d4b5d3be0588-istio-token\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-9r5c8\" (UID: \"83adee7e-3585-4d0d-b1b2-d4b5d3be0588\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-9r5c8"
Oct 30 22:10:01.102677 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:10:01.102661    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83adee7e-3585-4d0d-b1b2-d4b5d3be0588-istio-envoy\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-9r5c8\" (UID: \"83adee7e-3585-4d0d-b1b2-d4b5d3be0588\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-9r5c8"
Oct 30 22:10:01.102879 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:10:01.102864    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83adee7e-3585-4d0d-b1b2-d4b5d3be0588-istio-podinfo\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-9r5c8\" (UID: \"83adee7e-3585-4d0d-b1b2-d4b5d3be0588\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-9r5c8"
Oct 30 22:10:01.104905 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:10:01.100720    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83adee7e-3585-4d0d-b1b2-d4b5d3be0588-workload-certs\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-9r5c8\" (UID: \"83adee7e-3585-4d0d-b1b2-d4b5d3be0588\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-9r5c8"
Oct 30 22:10:01.107796 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:10:01.101174    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83adee7e-3585-4d0d-b1b2-d4b5d3be0588-credential-socket\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-9r5c8\" (UID: \"83adee7e-3585-4d0d-b1b2-d4b5d3be0588\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-9r5c8"
Oct 30 22:10:01.109798 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:10:01.109772    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83adee7e-3585-4d0d-b1b2-d4b5d3be0588-istiod-ca-cert\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-9r5c8\" (UID: \"83adee7e-3585-4d0d-b1b2-d4b5d3be0588\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-9r5c8"
Oct 30 22:10:01.110067 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:10:01.110044    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83adee7e-3585-4d0d-b1b2-d4b5d3be0588-workload-socket\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-9r5c8\" (UID: \"83adee7e-3585-4d0d-b1b2-d4b5d3be0588\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-9r5c8"
Oct 30 22:10:01.110912 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:10:01.110860    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83adee7e-3585-4d0d-b1b2-d4b5d3be0588-istio-data\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-9r5c8\" (UID: \"83adee7e-3585-4d0d-b1b2-d4b5d3be0588\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-9r5c8"
Oct 30 22:10:01.124475 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:10:01.124440    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83adee7e-3585-4d0d-b1b2-d4b5d3be0588-istio-podinfo\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-9r5c8\" (UID: \"83adee7e-3585-4d0d-b1b2-d4b5d3be0588\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-9r5c8"
Oct 30 22:10:01.160457 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:10:01.160385    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83adee7e-3585-4d0d-b1b2-d4b5d3be0588-istio-envoy\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-9r5c8\" (UID: \"83adee7e-3585-4d0d-b1b2-d4b5d3be0588\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-9r5c8"
Oct 30 22:10:01.163297 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:10:01.163219    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqs79\" (UniqueName: \"kubernetes.io/projected/83adee7e-3585-4d0d-b1b2-d4b5d3be0588-kube-api-access-gqs79\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-9r5c8\" (UID: \"83adee7e-3585-4d0d-b1b2-d4b5d3be0588\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-9r5c8"
Oct 30 22:10:01.170919 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:10:01.170889    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83adee7e-3585-4d0d-b1b2-d4b5d3be0588-istio-token\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-9r5c8\" (UID: \"83adee7e-3585-4d0d-b1b2-d4b5d3be0588\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-9r5c8"
Oct 30 22:10:01.203130 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:10:01.203094    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-9r5c8"
Oct 30 22:10:01.632209 ip-10-0-130-60 kubenswrapper[5061]: W1030 22:10:01.632182    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83adee7e_3585_4d0d_b1b2_d4b5d3be0588.slice/crio-311caf6c228b1dd09e6a7ae2a7478c8481c1993891207439288228ffc36c5123 WatchSource:0}: Error finding container 311caf6c228b1dd09e6a7ae2a7478c8481c1993891207439288228ffc36c5123: Status 404 returned error can't find the container with id 311caf6c228b1dd09e6a7ae2a7478c8481c1993891207439288228ffc36c5123
Oct 30 22:10:01.635147 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:10:01.635109    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:10:01.635249 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:10:01.635219    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:10:01.635292 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:10:01.635263    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:10:01.637629 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:10:01.637601    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-9r5c8"]
Oct 30 22:10:01.767378 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:10:01.767319    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-9r5c8" event={"ID":"83adee7e-3585-4d0d-b1b2-d4b5d3be0588","Type":"ContainerStarted","Data":"28d948b4308d1a2ef8191c64f52379ad5384f4efea11fb8817b380e34d76df2d"}
Oct 30 22:10:01.767600 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:10:01.767407    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-9r5c8" event={"ID":"83adee7e-3585-4d0d-b1b2-d4b5d3be0588","Type":"ContainerStarted","Data":"311caf6c228b1dd09e6a7ae2a7478c8481c1993891207439288228ffc36c5123"}
Oct 30 22:10:01.893388 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:10:01.893287    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-9r5c8" podStartSLOduration=1.893271634 podStartE2EDuration="1.893271634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:10:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:10:01.890541281 +0000 UTC m=+1655.503311049" watchObservedRunningTime="2026-10-30 22:10:01.893271634 +0000 UTC m=+1655.506041392"
Oct 30 22:10:02.203353 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:10:02.203276    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-9r5c8"
Oct 30 22:10:02.209238 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:10:02.209211    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-9r5c8"
Oct 30 22:10:02.774989 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:10:02.774958    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-9r5c8"
Oct 30 22:10:02.776014 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:10:02.775993    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-9r5c8"
Oct 30 22:10:11.475412 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:10:11.475302    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-778d39d3-5cc665b758-td28d"]
Oct 30 22:10:11.481512 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:10:11.481483    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-778d39d3-5cc665b758-td28d"
Oct 30 22:10:11.535550 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:10:11.535140    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-778d39d3-5cc665b758-td28d"]
Oct 30 22:10:11.582812 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:10:11.582778    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-778d39d3\""
Oct 30 22:10:11.592225 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:10:11.592199    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-778d39d3-dockercfg-gp7v5\""
Oct 30 22:10:11.604967 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:10:11.604924    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0abd91d-7465-4e7f-a840-827ddf311ce7-plugins-config-volume\") pod \"payload-processing-e2e-mt-778d39d3-5cc665b758-td28d\" (UID: \"b0abd91d-7465-4e7f-a840-827ddf311ce7\") " pod="openshift-ingress/payload-processing-e2e-mt-778d39d3-5cc665b758-td28d"
Oct 30 22:10:11.605602 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:10:11.605576    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9xtk\" (UniqueName: \"kubernetes.io/projected/b0abd91d-7465-4e7f-a840-827ddf311ce7-kube-api-access-g9xtk\") pod \"payload-processing-e2e-mt-778d39d3-5cc665b758-td28d\" (UID: \"b0abd91d-7465-4e7f-a840-827ddf311ce7\") " pod="openshift-ingress/payload-processing-e2e-mt-778d39d3-5cc665b758-td28d"
Oct 30 22:10:11.709699 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:10:11.709633    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0abd91d-7465-4e7f-a840-827ddf311ce7-plugins-config-volume\") pod \"payload-processing-e2e-mt-778d39d3-5cc665b758-td28d\" (UID: \"b0abd91d-7465-4e7f-a840-827ddf311ce7\") " pod="openshift-ingress/payload-processing-e2e-mt-778d39d3-5cc665b758-td28d"
Oct 30 22:10:11.709881 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:10:11.709818    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9xtk\" (UniqueName: \"kubernetes.io/projected/b0abd91d-7465-4e7f-a840-827ddf311ce7-kube-api-access-g9xtk\") pod \"payload-processing-e2e-mt-778d39d3-5cc665b758-td28d\" (UID: \"b0abd91d-7465-4e7f-a840-827ddf311ce7\") " pod="openshift-ingress/payload-processing-e2e-mt-778d39d3-5cc665b758-td28d"
Oct 30 22:10:11.711008 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:10:11.710988    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0abd91d-7465-4e7f-a840-827ddf311ce7-plugins-config-volume\") pod \"payload-processing-e2e-mt-778d39d3-5cc665b758-td28d\" (UID: \"b0abd91d-7465-4e7f-a840-827ddf311ce7\") " pod="openshift-ingress/payload-processing-e2e-mt-778d39d3-5cc665b758-td28d"
Oct 30 22:10:11.779514 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:10:11.779434    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9xtk\" (UniqueName: \"kubernetes.io/projected/b0abd91d-7465-4e7f-a840-827ddf311ce7-kube-api-access-g9xtk\") pod \"payload-processing-e2e-mt-778d39d3-5cc665b758-td28d\" (UID: \"b0abd91d-7465-4e7f-a840-827ddf311ce7\") " pod="openshift-ingress/payload-processing-e2e-mt-778d39d3-5cc665b758-td28d"
Oct 30 22:10:11.801887 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:10:11.801855    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-778d39d3-5cc665b758-td28d"
Oct 30 22:10:12.192321 ip-10-0-130-60 kubenswrapper[5061]: W1030 22:10:12.192287    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0abd91d_7465_4e7f_a840_827ddf311ce7.slice/crio-dc60efa1eb8fecc91e9e09dccc26c4a92585af62a34481165bd795fd4ca55b3f WatchSource:0}: Error finding container dc60efa1eb8fecc91e9e09dccc26c4a92585af62a34481165bd795fd4ca55b3f: Status 404 returned error can't find the container with id dc60efa1eb8fecc91e9e09dccc26c4a92585af62a34481165bd795fd4ca55b3f
Oct 30 22:10:12.197807 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:10:12.197778    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-778d39d3-5cc665b758-td28d"]
Oct 30 22:10:12.856141 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:10:12.856106    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-778d39d3-5cc665b758-td28d" event={"ID":"b0abd91d-7465-4e7f-a840-827ddf311ce7","Type":"ContainerStarted","Data":"f852069c4c6d8bce367216eb0e45333a23125648e02c584dbc7e2da2f868c00f"}
Oct 30 22:10:12.856672 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:10:12.856159    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-778d39d3-5cc665b758-td28d" event={"ID":"b0abd91d-7465-4e7f-a840-827ddf311ce7","Type":"ContainerStarted","Data":"dc60efa1eb8fecc91e9e09dccc26c4a92585af62a34481165bd795fd4ca55b3f"}
Oct 30 22:10:12.856672 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:10:12.856189    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-778d39d3-5cc665b758-td28d"
Oct 30 22:10:12.995230 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:10:12.995176    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-778d39d3-5cc665b758-td28d" podStartSLOduration=1.995163386 podStartE2EDuration="1.995163386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:10:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:10:12.991319412 +0000 UTC m=+1666.604089182" watchObservedRunningTime="2026-10-30 22:10:12.995163386 +0000 UTC m=+1666.607933144"
Oct 30 22:10:23.866981 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:10:23.866950    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-778d39d3-5cc665b758-td28d"
Oct 30 22:10:24.671890 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:10:24.671847    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 22:10:28.109351 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:10:28.109319    5061 scope.go:122] "RemoveContainer" containerID="d85ce38af7c029e19670695188b2d5eeb9c842425390a35b8e53692920adf996"
Oct 30 22:10:28.119868 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:10:28.119847    5061 scope.go:122] "RemoveContainer" containerID="1ecd168e1a486e07197222219256636047ce3ed4ee26a9edfa7b8b602101a9f5"
Oct 30 22:10:28.130335 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:10:28.130304    5061 scope.go:122] "RemoveContainer" containerID="79942dbff1f6f58c04e91105bf0ca84032a3f559815eef93a7e9d01221aa97d0"
Oct 30 22:10:28.139996 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:10:28.139977    5061 scope.go:122] "RemoveContainer" containerID="e96c9b2c27304c01e32242a5692a786ba88659d68eb97979ccee5f44c627e9a2"
Oct 30 22:10:28.149675 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:10:28.149653    5061 scope.go:122] "RemoveContainer" containerID="b1be1536ac5dea3754f7823c3c90d337ec4430cb02fa9b31c65f93a1a1dfd0cf"
Oct 30 22:10:54.668272 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:10:54.668228    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 22:10:57.194859 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:10:57.194822    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-kdmlq"]
Oct 30 22:10:57.195338 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:10:57.195067    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-kdmlq" podUID="1aec235a-1d74-4044-aa3d-bd87a6a2d86f" containerName="istio-proxy" containerID="cri-o://30a4ed5a6cb2e9a2845117356449fa90af9c06c206524fcf5bf3976b75992727" gracePeriod=30
Oct 30 22:10:57.247398 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:10:57.247368    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-9kcgf"]
Oct 30 22:10:57.247699 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:10:57.247667    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-9kcgf" podUID="9b521eca-5188-43b6-b589-d0ccab8c4520" containerName="istio-proxy" containerID="cri-o://bcb2e4028f11ad1fa672f8a687f6cbd4285f037f8856cfeb5e62b899b3f87113" gracePeriod=30
Oct 30 22:10:57.252164 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:10:57.252141    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-4j4xr"]
Oct 30 22:10:57.252318 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:10:57.252300    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-4j4xr" podUID="aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df" containerName="istio-proxy" containerID="cri-o://8b977b2c9b1fc9c6ae43d67d2948b12ffc7573a3cf71688b8bc9de22f1bf0fea" gracePeriod=30
Oct 30 22:10:57.299966 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:10:57.299929    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-kmtkh"]
Oct 30 22:10:57.300202 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:10:57.300179    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-kmtkh" podUID="03533dfe-8974-487d-9af6-4adda6fc8d8b" containerName="istio-proxy" containerID="cri-o://49633a346981edd93f36051e1be89b70e4aed785827492ede902087003443483" gracePeriod=30
Oct 30 22:10:57.304160 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:10:57.304137    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-rh8lc"]
Oct 30 22:10:57.304338 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:10:57.304306    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-rh8lc" podUID="d238b910-dd97-4901-8e16-274b063b66ec" containerName="istio-proxy" containerID="cri-o://a0d98cfa8080dbbca0afe6db33b277854ae57aea174343d33e4a310c664dc2aa" gracePeriod=30
Oct 30 22:10:57.377726 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:10:57.377690    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5zvwr"]
Oct 30 22:10:57.377914 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:10:57.377875    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5zvwr" podUID="3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf" containerName="istio-proxy" containerID="cri-o://bb5f3d13f979cbf2216cd47e9e0c6af992a4f3cda415ac5fd12663e716e9d48c" gracePeriod=30
Oct 30 22:11:01.456483 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:01.456431    5061 patch_prober.go:29] interesting pod/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5zvwr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:11:01.458265 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:01.458229    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5zvwr" podUID="3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:11:01.665377 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:01.665341    5061 patch_prober.go:29] interesting pod/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-rh8lc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:11:01.665556 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:01.665417    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-rh8lc" podUID="d238b910-dd97-4901-8e16-274b063b66ec" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:11:02.494348 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:02.470845    5061 generic.go:356] "Generic (PLEG): container finished" podID="9b521eca-5188-43b6-b589-d0ccab8c4520" containerID="bcb2e4028f11ad1fa672f8a687f6cbd4285f037f8856cfeb5e62b899b3f87113" exitCode=0
Oct 30 22:11:02.494348 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:02.470946    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-9kcgf" event={"ID":"9b521eca-5188-43b6-b589-d0ccab8c4520","Type":"ContainerDied","Data":"bcb2e4028f11ad1fa672f8a687f6cbd4285f037f8856cfeb5e62b899b3f87113"}
Oct 30 22:11:02.494348 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:02.472929    5061 generic.go:356] "Generic (PLEG): container finished" podID="d238b910-dd97-4901-8e16-274b063b66ec" containerID="a0d98cfa8080dbbca0afe6db33b277854ae57aea174343d33e4a310c664dc2aa" exitCode=0
Oct 30 22:11:02.494348 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:02.473113    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-rh8lc" event={"ID":"d238b910-dd97-4901-8e16-274b063b66ec","Type":"ContainerDied","Data":"a0d98cfa8080dbbca0afe6db33b277854ae57aea174343d33e4a310c664dc2aa"}
Oct 30 22:11:02.494348 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:02.485728    5061 generic.go:356] "Generic (PLEG): container finished" podID="03533dfe-8974-487d-9af6-4adda6fc8d8b" containerID="49633a346981edd93f36051e1be89b70e4aed785827492ede902087003443483" exitCode=0
Oct 30 22:11:02.494348 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:02.485855    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-kmtkh" event={"ID":"03533dfe-8974-487d-9af6-4adda6fc8d8b","Type":"ContainerDied","Data":"49633a346981edd93f36051e1be89b70e4aed785827492ede902087003443483"}
Oct 30 22:11:02.494348 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:02.488105    5061 generic.go:356] "Generic (PLEG): container finished" podID="1aec235a-1d74-4044-aa3d-bd87a6a2d86f" containerID="30a4ed5a6cb2e9a2845117356449fa90af9c06c206524fcf5bf3976b75992727" exitCode=0
Oct 30 22:11:02.494348 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:02.488197    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-kdmlq" event={"ID":"1aec235a-1d74-4044-aa3d-bd87a6a2d86f","Type":"ContainerDied","Data":"30a4ed5a6cb2e9a2845117356449fa90af9c06c206524fcf5bf3976b75992727"}
Oct 30 22:11:02.494348 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:02.491407    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-4j4xr" event={"ID":"aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df","Type":"ContainerDied","Data":"8b977b2c9b1fc9c6ae43d67d2948b12ffc7573a3cf71688b8bc9de22f1bf0fea"}
Oct 30 22:11:02.494348 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:02.490602    5061 generic.go:356] "Generic (PLEG): container finished" podID="aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df" containerID="8b977b2c9b1fc9c6ae43d67d2948b12ffc7573a3cf71688b8bc9de22f1bf0fea" exitCode=0
Oct 30 22:11:02.724250 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:02.724221    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-kdmlq"
Oct 30 22:11:02.840068 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:02.840039    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-9kcgf"
Oct 30 22:11:02.842686 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:02.842650    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1aec235a-1d74-4044-aa3d-bd87a6a2d86f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1aec235a-1d74-4044-aa3d-bd87a6a2d86f-istio-envoy\") pod \"1aec235a-1d74-4044-aa3d-bd87a6a2d86f\" (UID: \"1aec235a-1d74-4044-aa3d-bd87a6a2d86f\") "
Oct 30 22:11:02.842844 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:02.842771    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1aec235a-1d74-4044-aa3d-bd87a6a2d86f-istio-token\" (UniqueName: \"kubernetes.io/projected/1aec235a-1d74-4044-aa3d-bd87a6a2d86f-istio-token\") pod \"1aec235a-1d74-4044-aa3d-bd87a6a2d86f\" (UID: \"1aec235a-1d74-4044-aa3d-bd87a6a2d86f\") "
Oct 30 22:11:02.842844 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:02.842839    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1aec235a-1d74-4044-aa3d-bd87a6a2d86f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aec235a-1d74-4044-aa3d-bd87a6a2d86f-workload-socket\") pod \"1aec235a-1d74-4044-aa3d-bd87a6a2d86f\" (UID: \"1aec235a-1d74-4044-aa3d-bd87a6a2d86f\") "
Oct 30 22:11:02.842972 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:02.842890    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1aec235a-1d74-4044-aa3d-bd87a6a2d86f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1aec235a-1d74-4044-aa3d-bd87a6a2d86f-istio-podinfo\") pod \"1aec235a-1d74-4044-aa3d-bd87a6a2d86f\" (UID: \"1aec235a-1d74-4044-aa3d-bd87a6a2d86f\") "
Oct 30 22:11:02.842972 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:02.842961    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1aec235a-1d74-4044-aa3d-bd87a6a2d86f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1aec235a-1d74-4044-aa3d-bd87a6a2d86f-istio-data\") pod \"1aec235a-1d74-4044-aa3d-bd87a6a2d86f\" (UID: \"1aec235a-1d74-4044-aa3d-bd87a6a2d86f\") "
Oct 30 22:11:02.843086 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:02.843047    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1aec235a-1d74-4044-aa3d-bd87a6a2d86f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aec235a-1d74-4044-aa3d-bd87a6a2d86f-credential-socket\") pod \"1aec235a-1d74-4044-aa3d-bd87a6a2d86f\" (UID: \"1aec235a-1d74-4044-aa3d-bd87a6a2d86f\") "
Oct 30 22:11:02.843146 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:02.843091    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1aec235a-1d74-4044-aa3d-bd87a6a2d86f-kube-api-access-86ngg\" (UniqueName: \"kubernetes.io/projected/1aec235a-1d74-4044-aa3d-bd87a6a2d86f-kube-api-access-86ngg\") pod \"1aec235a-1d74-4044-aa3d-bd87a6a2d86f\" (UID: \"1aec235a-1d74-4044-aa3d-bd87a6a2d86f\") "
Oct 30 22:11:02.843231 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:02.843213    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1aec235a-1d74-4044-aa3d-bd87a6a2d86f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1aec235a-1d74-4044-aa3d-bd87a6a2d86f-workload-certs\") pod \"1aec235a-1d74-4044-aa3d-bd87a6a2d86f\" (UID: \"1aec235a-1d74-4044-aa3d-bd87a6a2d86f\") "
Oct 30 22:11:02.843285 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:02.843274    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1aec235a-1d74-4044-aa3d-bd87a6a2d86f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1aec235a-1d74-4044-aa3d-bd87a6a2d86f-istiod-ca-cert\") pod \"1aec235a-1d74-4044-aa3d-bd87a6a2d86f\" (UID: \"1aec235a-1d74-4044-aa3d-bd87a6a2d86f\") "
Oct 30 22:11:02.844726 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:02.844416    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1aec235a-1d74-4044-aa3d-bd87a6a2d86f-istiod-ca-cert" pod "1aec235a-1d74-4044-aa3d-bd87a6a2d86f" (UID: "1aec235a-1d74-4044-aa3d-bd87a6a2d86f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:11:02.846700 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:02.846673    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1aec235a-1d74-4044-aa3d-bd87a6a2d86f-istio-data" pod "1aec235a-1d74-4044-aa3d-bd87a6a2d86f" (UID: "1aec235a-1d74-4044-aa3d-bd87a6a2d86f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:02.847610 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:02.847491    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1aec235a-1d74-4044-aa3d-bd87a6a2d86f-credential-socket" pod "1aec235a-1d74-4044-aa3d-bd87a6a2d86f" (UID: "1aec235a-1d74-4044-aa3d-bd87a6a2d86f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:02.847610 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:02.847543    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1aec235a-1d74-4044-aa3d-bd87a6a2d86f-workload-certs" pod "1aec235a-1d74-4044-aa3d-bd87a6a2d86f" (UID: "1aec235a-1d74-4044-aa3d-bd87a6a2d86f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:02.847610 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:02.847590    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1aec235a-1d74-4044-aa3d-bd87a6a2d86f-workload-socket" pod "1aec235a-1d74-4044-aa3d-bd87a6a2d86f" (UID: "1aec235a-1d74-4044-aa3d-bd87a6a2d86f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:02.848644 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:02.848609    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1aec235a-1d74-4044-aa3d-bd87a6a2d86f-istio-envoy" pod "1aec235a-1d74-4044-aa3d-bd87a6a2d86f" (UID: "1aec235a-1d74-4044-aa3d-bd87a6a2d86f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:02.859219 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:02.859153    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1aec235a-1d74-4044-aa3d-bd87a6a2d86f-istio-podinfo" pod "1aec235a-1d74-4044-aa3d-bd87a6a2d86f" (UID: "1aec235a-1d74-4044-aa3d-bd87a6a2d86f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:11:02.859219 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:02.859160    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1aec235a-1d74-4044-aa3d-bd87a6a2d86f-istio-token" pod "1aec235a-1d74-4044-aa3d-bd87a6a2d86f" (UID: "1aec235a-1d74-4044-aa3d-bd87a6a2d86f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:11:02.866890 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:02.866851    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1aec235a-1d74-4044-aa3d-bd87a6a2d86f-kube-api-access-86ngg" pod "1aec235a-1d74-4044-aa3d-bd87a6a2d86f" (UID: "1aec235a-1d74-4044-aa3d-bd87a6a2d86f"). InnerVolumeSpecName "kube-api-access-86ngg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:11:02.934086 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:02.934061    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-kmtkh"
Oct 30 22:11:02.936404 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:02.936384    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-4j4xr"
Oct 30 22:11:02.941666 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:02.941642    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-rh8lc"
Oct 30 22:11:02.944774 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:02.944751    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b521eca-5188-43b6-b589-d0ccab8c4520-kube-api-access-g4qls\" (UniqueName: \"kubernetes.io/projected/9b521eca-5188-43b6-b589-d0ccab8c4520-kube-api-access-g4qls\") pod \"9b521eca-5188-43b6-b589-d0ccab8c4520\" (UID: \"9b521eca-5188-43b6-b589-d0ccab8c4520\") "
Oct 30 22:11:02.944902 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:02.944832    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b521eca-5188-43b6-b589-d0ccab8c4520-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b521eca-5188-43b6-b589-d0ccab8c4520-credential-socket\") pod \"9b521eca-5188-43b6-b589-d0ccab8c4520\" (UID: \"9b521eca-5188-43b6-b589-d0ccab8c4520\") "
Oct 30 22:11:02.944902 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:02.944887    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b521eca-5188-43b6-b589-d0ccab8c4520-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b521eca-5188-43b6-b589-d0ccab8c4520-workload-socket\") pod \"9b521eca-5188-43b6-b589-d0ccab8c4520\" (UID: \"9b521eca-5188-43b6-b589-d0ccab8c4520\") "
Oct 30 22:11:02.945007 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:02.944918    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b521eca-5188-43b6-b589-d0ccab8c4520-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b521eca-5188-43b6-b589-d0ccab8c4520-istio-envoy\") pod \"9b521eca-5188-43b6-b589-d0ccab8c4520\" (UID: \"9b521eca-5188-43b6-b589-d0ccab8c4520\") "
Oct 30 22:11:02.945007 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:02.944977    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b521eca-5188-43b6-b589-d0ccab8c4520-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b521eca-5188-43b6-b589-d0ccab8c4520-workload-certs\") pod \"9b521eca-5188-43b6-b589-d0ccab8c4520\" (UID: \"9b521eca-5188-43b6-b589-d0ccab8c4520\") "
Oct 30 22:11:02.945114 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:02.945059    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b521eca-5188-43b6-b589-d0ccab8c4520-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b521eca-5188-43b6-b589-d0ccab8c4520-istiod-ca-cert\") pod \"9b521eca-5188-43b6-b589-d0ccab8c4520\" (UID: \"9b521eca-5188-43b6-b589-d0ccab8c4520\") "
Oct 30 22:11:02.945170 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:02.945139    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9b521eca-5188-43b6-b589-d0ccab8c4520-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b521eca-5188-43b6-b589-d0ccab8c4520-istio-podinfo\") pod \"9b521eca-5188-43b6-b589-d0ccab8c4520\" (UID: \"9b521eca-5188-43b6-b589-d0ccab8c4520\") "
Oct 30 22:11:02.945407 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:02.945255    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b521eca-5188-43b6-b589-d0ccab8c4520-istio-token\" (UniqueName: \"kubernetes.io/projected/9b521eca-5188-43b6-b589-d0ccab8c4520-istio-token\") pod \"9b521eca-5188-43b6-b589-d0ccab8c4520\" (UID: \"9b521eca-5188-43b6-b589-d0ccab8c4520\") "
Oct 30 22:11:02.945407 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:02.945286    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b521eca-5188-43b6-b589-d0ccab8c4520-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b521eca-5188-43b6-b589-d0ccab8c4520-istio-data\") pod \"9b521eca-5188-43b6-b589-d0ccab8c4520\" (UID: \"9b521eca-5188-43b6-b589-d0ccab8c4520\") "
Oct 30 22:11:02.945787 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:02.945604    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b521eca-5188-43b6-b589-d0ccab8c4520-workload-socket" pod "9b521eca-5188-43b6-b589-d0ccab8c4520" (UID: "9b521eca-5188-43b6-b589-d0ccab8c4520"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:02.945866 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:02.945814    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b521eca-5188-43b6-b589-d0ccab8c4520-istio-data" pod "9b521eca-5188-43b6-b589-d0ccab8c4520" (UID: "9b521eca-5188-43b6-b589-d0ccab8c4520"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:02.946236 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:02.946210    5061 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1aec235a-1d74-4044-aa3d-bd87a6a2d86f-istio-envoy\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:02.946478 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:02.946240    5061 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b521eca-5188-43b6-b589-d0ccab8c4520-istio-data\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:02.946478 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:02.946255    5061 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1aec235a-1d74-4044-aa3d-bd87a6a2d86f-istio-token\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:02.946478 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:02.946270    5061 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aec235a-1d74-4044-aa3d-bd87a6a2d86f-workload-socket\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:02.946478 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:02.946274    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5zvwr"
Oct 30 22:11:02.946478 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:02.946284    5061 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1aec235a-1d74-4044-aa3d-bd87a6a2d86f-istio-podinfo\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:02.946478 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:02.946298    5061 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1aec235a-1d74-4044-aa3d-bd87a6a2d86f-istio-data\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:02.946478 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:02.946312    5061 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b521eca-5188-43b6-b589-d0ccab8c4520-workload-socket\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:02.946478 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:02.946336    5061 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aec235a-1d74-4044-aa3d-bd87a6a2d86f-credential-socket\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:02.946478 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:02.946350    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86ngg\" (UniqueName: \"kubernetes.io/projected/1aec235a-1d74-4044-aa3d-bd87a6a2d86f-kube-api-access-86ngg\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:02.946478 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:02.946366    5061 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1aec235a-1d74-4044-aa3d-bd87a6a2d86f-workload-certs\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:02.946478 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:02.946379    5061 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1aec235a-1d74-4044-aa3d-bd87a6a2d86f-istiod-ca-cert\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:02.947372 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:02.946702    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b521eca-5188-43b6-b589-d0ccab8c4520-istiod-ca-cert" pod "9b521eca-5188-43b6-b589-d0ccab8c4520" (UID: "9b521eca-5188-43b6-b589-d0ccab8c4520"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:11:02.947372 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:02.946748    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b521eca-5188-43b6-b589-d0ccab8c4520-workload-certs" pod "9b521eca-5188-43b6-b589-d0ccab8c4520" (UID: "9b521eca-5188-43b6-b589-d0ccab8c4520"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:02.947372 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:02.946808    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b521eca-5188-43b6-b589-d0ccab8c4520-credential-socket" pod "9b521eca-5188-43b6-b589-d0ccab8c4520" (UID: "9b521eca-5188-43b6-b589-d0ccab8c4520"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:02.948758 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:02.948733    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b521eca-5188-43b6-b589-d0ccab8c4520-kube-api-access-g4qls" pod "9b521eca-5188-43b6-b589-d0ccab8c4520" (UID: "9b521eca-5188-43b6-b589-d0ccab8c4520"). InnerVolumeSpecName "kube-api-access-g4qls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:11:02.949889 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:02.949848    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9b521eca-5188-43b6-b589-d0ccab8c4520-istio-podinfo" pod "9b521eca-5188-43b6-b589-d0ccab8c4520" (UID: "9b521eca-5188-43b6-b589-d0ccab8c4520"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:11:02.949889 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:02.949857    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b521eca-5188-43b6-b589-d0ccab8c4520-istio-envoy" pod "9b521eca-5188-43b6-b589-d0ccab8c4520" (UID: "9b521eca-5188-43b6-b589-d0ccab8c4520"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:02.954785 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:02.953706    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b521eca-5188-43b6-b589-d0ccab8c4520-istio-token" pod "9b521eca-5188-43b6-b589-d0ccab8c4520" (UID: "9b521eca-5188-43b6-b589-d0ccab8c4520"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:11:03.047497 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.047462    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d238b910-dd97-4901-8e16-274b063b66ec-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d238b910-dd97-4901-8e16-274b063b66ec-istio-envoy\") pod \"d238b910-dd97-4901-8e16-274b063b66ec\" (UID: \"d238b910-dd97-4901-8e16-274b063b66ec\") "
Oct 30 22:11:03.047801 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.047770    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf-workload-socket\") pod \"3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf\" (UID: \"3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf\") "
Oct 30 22:11:03.047935 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.047831    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03533dfe-8974-487d-9af6-4adda6fc8d8b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03533dfe-8974-487d-9af6-4adda6fc8d8b-workload-socket\") pod \"03533dfe-8974-487d-9af6-4adda6fc8d8b\" (UID: \"03533dfe-8974-487d-9af6-4adda6fc8d8b\") "
Oct 30 22:11:03.047935 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.047879    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df-istio-podinfo\") pod \"aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df\" (UID: \"aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df\") "
Oct 30 22:11:03.047935 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.047907    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df-istiod-ca-cert\") pod \"aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df\" (UID: \"aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df\") "
Oct 30 22:11:03.048308 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.047950    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf-istio-data\") pod \"3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf\" (UID: \"3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf\") "
Oct 30 22:11:03.048308 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.047985    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df-credential-socket\") pod \"aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df\" (UID: \"aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df\") "
Oct 30 22:11:03.048308 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.048036    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df-workload-socket\") pod \"aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df\" (UID: \"aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df\") "
Oct 30 22:11:03.048308 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.048068    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d238b910-dd97-4901-8e16-274b063b66ec-istio-token\" (UniqueName: \"kubernetes.io/projected/d238b910-dd97-4901-8e16-274b063b66ec-istio-token\") pod \"d238b910-dd97-4901-8e16-274b063b66ec\" (UID: \"d238b910-dd97-4901-8e16-274b063b66ec\") "
Oct 30 22:11:03.048308 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.048117    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03533dfe-8974-487d-9af6-4adda6fc8d8b-workload-socket" pod "03533dfe-8974-487d-9af6-4adda6fc8d8b" (UID: "03533dfe-8974-487d-9af6-4adda6fc8d8b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:03.048308 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.048138    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03533dfe-8974-487d-9af6-4adda6fc8d8b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03533dfe-8974-487d-9af6-4adda6fc8d8b-istiod-ca-cert\") pod \"03533dfe-8974-487d-9af6-4adda6fc8d8b\" (UID: \"03533dfe-8974-487d-9af6-4adda6fc8d8b\") "
Oct 30 22:11:03.048742 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.048457    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df-credential-socket" pod "aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df" (UID: "aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:03.048742 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.048479    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf-istio-data" pod "3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf" (UID: "3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:03.048742 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.048723    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d238b910-dd97-4901-8e16-274b063b66ec-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d238b910-dd97-4901-8e16-274b063b66ec-workload-socket\") pod \"d238b910-dd97-4901-8e16-274b063b66ec\" (UID: \"d238b910-dd97-4901-8e16-274b063b66ec\") "
Oct 30 22:11:03.048900 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.048779    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf-istio-envoy\") pod \"3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf\" (UID: \"3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf\") "
Oct 30 22:11:03.048900 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.048809    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d238b910-dd97-4901-8e16-274b063b66ec-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d238b910-dd97-4901-8e16-274b063b66ec-workload-certs\") pod \"d238b910-dd97-4901-8e16-274b063b66ec\" (UID: \"d238b910-dd97-4901-8e16-274b063b66ec\") "
Oct 30 22:11:03.048900 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.048825    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df-istiod-ca-cert" pod "aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df" (UID: "aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:11:03.048900 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.048847    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03533dfe-8974-487d-9af6-4adda6fc8d8b-kube-api-access-vw74n\" (UniqueName: \"kubernetes.io/projected/03533dfe-8974-487d-9af6-4adda6fc8d8b-kube-api-access-vw74n\") pod \"03533dfe-8974-487d-9af6-4adda6fc8d8b\" (UID: \"03533dfe-8974-487d-9af6-4adda6fc8d8b\") "
Oct 30 22:11:03.049126 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.048925    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d238b910-dd97-4901-8e16-274b063b66ec-kube-api-access-krfrm\" (UniqueName: \"kubernetes.io/projected/d238b910-dd97-4901-8e16-274b063b66ec-kube-api-access-krfrm\") pod \"d238b910-dd97-4901-8e16-274b063b66ec\" (UID: \"d238b910-dd97-4901-8e16-274b063b66ec\") "
Oct 30 22:11:03.049126 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.048962    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf-istiod-ca-cert\") pod \"3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf\" (UID: \"3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf\") "
Oct 30 22:11:03.049126 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.049025    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d238b910-dd97-4901-8e16-274b063b66ec-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d238b910-dd97-4901-8e16-274b063b66ec-credential-socket\") pod \"d238b910-dd97-4901-8e16-274b063b66ec\" (UID: \"d238b910-dd97-4901-8e16-274b063b66ec\") "
Oct 30 22:11:03.049126 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.049060    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d238b910-dd97-4901-8e16-274b063b66ec-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d238b910-dd97-4901-8e16-274b063b66ec-istiod-ca-cert\") pod \"d238b910-dd97-4901-8e16-274b063b66ec\" (UID: \"d238b910-dd97-4901-8e16-274b063b66ec\") "
Oct 30 22:11:03.049126 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.049096    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/03533dfe-8974-487d-9af6-4adda6fc8d8b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03533dfe-8974-487d-9af6-4adda6fc8d8b-istio-podinfo\") pod \"03533dfe-8974-487d-9af6-4adda6fc8d8b\" (UID: \"03533dfe-8974-487d-9af6-4adda6fc8d8b\") "
Oct 30 22:11:03.049379 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.049137    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf-workload-certs\") pod \"3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf\" (UID: \"3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf\") "
Oct 30 22:11:03.049379 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.049169    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03533dfe-8974-487d-9af6-4adda6fc8d8b-istio-token\" (UniqueName: \"kubernetes.io/projected/03533dfe-8974-487d-9af6-4adda6fc8d8b-istio-token\") pod \"03533dfe-8974-487d-9af6-4adda6fc8d8b\" (UID: \"03533dfe-8974-487d-9af6-4adda6fc8d8b\") "
Oct 30 22:11:03.049379 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.049203    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf-istio-token\" (UniqueName: \"kubernetes.io/projected/3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf-istio-token\") pod \"3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf\" (UID: \"3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf\") "
Oct 30 22:11:03.049379 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.049267    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df-istio-data\") pod \"aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df\" (UID: \"aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df\") "
Oct 30 22:11:03.049379 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.049298    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df-kube-api-access-4wg6m\" (UniqueName: \"kubernetes.io/projected/aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df-kube-api-access-4wg6m\") pod \"aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df\" (UID: \"aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df\") "
Oct 30 22:11:03.049379 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.049326    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df-workload-certs\") pod \"aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df\" (UID: \"aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df\") "
Oct 30 22:11:03.049379 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.049324    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf-workload-socket" pod "3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf" (UID: "3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:03.049766 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.049382    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf-credential-socket\") pod \"3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf\" (UID: \"3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf\") "
Oct 30 22:11:03.049766 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.049431    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03533dfe-8974-487d-9af6-4adda6fc8d8b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03533dfe-8974-487d-9af6-4adda6fc8d8b-istio-envoy\") pod \"03533dfe-8974-487d-9af6-4adda6fc8d8b\" (UID: \"03533dfe-8974-487d-9af6-4adda6fc8d8b\") "
Oct 30 22:11:03.049766 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.049473    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df-istio-envoy\") pod \"aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df\" (UID: \"aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df\") "
Oct 30 22:11:03.049766 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.049516    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03533dfe-8974-487d-9af6-4adda6fc8d8b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03533dfe-8974-487d-9af6-4adda6fc8d8b-istio-data\") pod \"03533dfe-8974-487d-9af6-4adda6fc8d8b\" (UID: \"03533dfe-8974-487d-9af6-4adda6fc8d8b\") "
Oct 30 22:11:03.049766 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.049587    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d238b910-dd97-4901-8e16-274b063b66ec-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d238b910-dd97-4901-8e16-274b063b66ec-istio-podinfo\") pod \"d238b910-dd97-4901-8e16-274b063b66ec\" (UID: \"d238b910-dd97-4901-8e16-274b063b66ec\") "
Oct 30 22:11:03.049766 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.049625    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d238b910-dd97-4901-8e16-274b063b66ec-workload-socket" pod "d238b910-dd97-4901-8e16-274b063b66ec" (UID: "d238b910-dd97-4901-8e16-274b063b66ec"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:03.049766 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.049637    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf-kube-api-access-7ht4x\" (UniqueName: \"kubernetes.io/projected/3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf-kube-api-access-7ht4x\") pod \"3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf\" (UID: \"3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf\") "
Oct 30 22:11:03.049766 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.049706    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df-istio-token\" (UniqueName: \"kubernetes.io/projected/aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df-istio-token\") pod \"aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df\" (UID: \"aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df\") "
Oct 30 22:11:03.050167 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.049802    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d238b910-dd97-4901-8e16-274b063b66ec-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d238b910-dd97-4901-8e16-274b063b66ec-istio-data\") pod \"d238b910-dd97-4901-8e16-274b063b66ec\" (UID: \"d238b910-dd97-4901-8e16-274b063b66ec\") "
Oct 30 22:11:03.050167 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.049834    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03533dfe-8974-487d-9af6-4adda6fc8d8b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03533dfe-8974-487d-9af6-4adda6fc8d8b-credential-socket\") pod \"03533dfe-8974-487d-9af6-4adda6fc8d8b\" (UID: \"03533dfe-8974-487d-9af6-4adda6fc8d8b\") "
Oct 30 22:11:03.050167 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.049876    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf-istio-podinfo\") pod \"3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf\" (UID: \"3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf\") "
Oct 30 22:11:03.050167 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.049937    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03533dfe-8974-487d-9af6-4adda6fc8d8b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03533dfe-8974-487d-9af6-4adda6fc8d8b-workload-certs\") pod \"03533dfe-8974-487d-9af6-4adda6fc8d8b\" (UID: \"03533dfe-8974-487d-9af6-4adda6fc8d8b\") "
Oct 30 22:11:03.050898 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.050863    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d238b910-dd97-4901-8e16-274b063b66ec-istio-envoy" pod "d238b910-dd97-4901-8e16-274b063b66ec" (UID: "d238b910-dd97-4901-8e16-274b063b66ec"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:03.051003 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.050946    5061 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b521eca-5188-43b6-b589-d0ccab8c4520-credential-socket\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:03.051003 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.050970    5061 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b521eca-5188-43b6-b589-d0ccab8c4520-istio-envoy\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:03.051003 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.050984    5061 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b521eca-5188-43b6-b589-d0ccab8c4520-workload-certs\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:03.051003 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.050997    5061 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b521eca-5188-43b6-b589-d0ccab8c4520-istiod-ca-cert\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:03.051303 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.051011    5061 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf-workload-socket\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:03.051303 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.051025    5061 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b521eca-5188-43b6-b589-d0ccab8c4520-istio-podinfo\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:03.051303 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.051039    5061 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03533dfe-8974-487d-9af6-4adda6fc8d8b-workload-socket\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:03.051303 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.051053    5061 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df-istiod-ca-cert\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:03.051303 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.051066    5061 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf-istio-data\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:03.051303 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.051081    5061 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df-credential-socket\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:03.051303 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.051097    5061 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b521eca-5188-43b6-b589-d0ccab8c4520-istio-token\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:03.051303 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.051110    5061 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d238b910-dd97-4901-8e16-274b063b66ec-workload-socket\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:03.051303 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.051125    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4qls\" (UniqueName: \"kubernetes.io/projected/9b521eca-5188-43b6-b589-d0ccab8c4520-kube-api-access-g4qls\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:03.051865 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.051334    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d238b910-dd97-4901-8e16-274b063b66ec-istio-data" pod "d238b910-dd97-4901-8e16-274b063b66ec" (UID: "d238b910-dd97-4901-8e16-274b063b66ec"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:03.051865 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.051364    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df-istio-podinfo" pod "aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df" (UID: "aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:11:03.051865 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.051421    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df-workload-socket" pod "aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df" (UID: "aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:03.051865 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.051437    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf-istiod-ca-cert" pod "3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf" (UID: "3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:11:03.051865 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.051740    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d238b910-dd97-4901-8e16-274b063b66ec-workload-certs" pod "d238b910-dd97-4901-8e16-274b063b66ec" (UID: "d238b910-dd97-4901-8e16-274b063b66ec"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:03.053030 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.052689    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03533dfe-8974-487d-9af6-4adda6fc8d8b-credential-socket" pod "03533dfe-8974-487d-9af6-4adda6fc8d8b" (UID: "03533dfe-8974-487d-9af6-4adda6fc8d8b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:03.053125 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.053019    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d238b910-dd97-4901-8e16-274b063b66ec-credential-socket" pod "d238b910-dd97-4901-8e16-274b063b66ec" (UID: "d238b910-dd97-4901-8e16-274b063b66ec"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:03.053324 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.053300    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d238b910-dd97-4901-8e16-274b063b66ec-istiod-ca-cert" pod "d238b910-dd97-4901-8e16-274b063b66ec" (UID: "d238b910-dd97-4901-8e16-274b063b66ec"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:11:03.054021 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.053985    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03533dfe-8974-487d-9af6-4adda6fc8d8b-istiod-ca-cert" pod "03533dfe-8974-487d-9af6-4adda6fc8d8b" (UID: "03533dfe-8974-487d-9af6-4adda6fc8d8b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:11:03.056049 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.054588    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03533dfe-8974-487d-9af6-4adda6fc8d8b-istio-data" pod "03533dfe-8974-487d-9af6-4adda6fc8d8b" (UID: "03533dfe-8974-487d-9af6-4adda6fc8d8b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:03.056049 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.054829    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df-workload-certs" pod "aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df" (UID: "aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:03.056049 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.055058    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df-istio-data" pod "aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df" (UID: "aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:03.056049 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.055567    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf-workload-certs" pod "3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf" (UID: "3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:03.056049 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.055779    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf-credential-socket" pod "3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf" (UID: "3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:03.056463 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.056110    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df-istio-envoy" pod "aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df" (UID: "aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:03.056463 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.056299    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03533dfe-8974-487d-9af6-4adda6fc8d8b-workload-certs" pod "03533dfe-8974-487d-9af6-4adda6fc8d8b" (UID: "03533dfe-8974-487d-9af6-4adda6fc8d8b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:03.057137 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.057094    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/03533dfe-8974-487d-9af6-4adda6fc8d8b-istio-podinfo" pod "03533dfe-8974-487d-9af6-4adda6fc8d8b" (UID: "03533dfe-8974-487d-9af6-4adda6fc8d8b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:11:03.060197 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.060142    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf-istio-podinfo" pod "3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf" (UID: "3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:11:03.060576 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.060506    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03533dfe-8974-487d-9af6-4adda6fc8d8b-istio-envoy" pod "03533dfe-8974-487d-9af6-4adda6fc8d8b" (UID: "03533dfe-8974-487d-9af6-4adda6fc8d8b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:03.060851 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.060828    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf-kube-api-access-7ht4x" pod "3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf" (UID: "3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf"). InnerVolumeSpecName "kube-api-access-7ht4x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:11:03.060938 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.060871    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03533dfe-8974-487d-9af6-4adda6fc8d8b-kube-api-access-vw74n" pod "03533dfe-8974-487d-9af6-4adda6fc8d8b" (UID: "03533dfe-8974-487d-9af6-4adda6fc8d8b"). InnerVolumeSpecName "kube-api-access-vw74n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:11:03.061031 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.061012    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d238b910-dd97-4901-8e16-274b063b66ec-istio-podinfo" pod "d238b910-dd97-4901-8e16-274b063b66ec" (UID: "d238b910-dd97-4901-8e16-274b063b66ec"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:11:03.061189 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.061164    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df-istio-token" pod "aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df" (UID: "aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:11:03.061325 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.061199    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03533dfe-8974-487d-9af6-4adda6fc8d8b-istio-token" pod "03533dfe-8974-487d-9af6-4adda6fc8d8b" (UID: "03533dfe-8974-487d-9af6-4adda6fc8d8b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:11:03.061506 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.061474    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d238b910-dd97-4901-8e16-274b063b66ec-kube-api-access-krfrm" pod "d238b910-dd97-4901-8e16-274b063b66ec" (UID: "d238b910-dd97-4901-8e16-274b063b66ec"). InnerVolumeSpecName "kube-api-access-krfrm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:11:03.061641 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.061576    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df-kube-api-access-4wg6m" pod "aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df" (UID: "aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df"). InnerVolumeSpecName "kube-api-access-4wg6m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:11:03.061897 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.061876    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d238b910-dd97-4901-8e16-274b063b66ec-istio-token" pod "d238b910-dd97-4901-8e16-274b063b66ec" (UID: "d238b910-dd97-4901-8e16-274b063b66ec"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:11:03.063020 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.062984    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf-istio-token" pod "3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf" (UID: "3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:11:03.063268 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.063245    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf-istio-envoy" pod "3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf" (UID: "3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:03.155137 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.155095    5061 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df-istio-podinfo\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:03.155137 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.155136    5061 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df-workload-socket\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:03.155404 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.155150    5061 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d238b910-dd97-4901-8e16-274b063b66ec-istio-token\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:03.155404 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.155162    5061 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03533dfe-8974-487d-9af6-4adda6fc8d8b-istiod-ca-cert\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:03.155404 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.155176    5061 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf-istio-envoy\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:03.155404 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.155189    5061 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d238b910-dd97-4901-8e16-274b063b66ec-workload-certs\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:03.155404 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.155203    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vw74n\" (UniqueName: \"kubernetes.io/projected/03533dfe-8974-487d-9af6-4adda6fc8d8b-kube-api-access-vw74n\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:03.155404 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.155217    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krfrm\" (UniqueName: \"kubernetes.io/projected/d238b910-dd97-4901-8e16-274b063b66ec-kube-api-access-krfrm\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:03.155404 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.155229    5061 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf-istiod-ca-cert\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:03.155404 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.155242    5061 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d238b910-dd97-4901-8e16-274b063b66ec-credential-socket\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:03.155404 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.155254    5061 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d238b910-dd97-4901-8e16-274b063b66ec-istiod-ca-cert\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:03.155404 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.155269    5061 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03533dfe-8974-487d-9af6-4adda6fc8d8b-istio-podinfo\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:03.155404 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.155293    5061 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf-workload-certs\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:03.155404 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.155305    5061 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03533dfe-8974-487d-9af6-4adda6fc8d8b-istio-token\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:03.155404 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.155316    5061 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf-istio-token\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:03.155404 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.155328    5061 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df-istio-data\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:03.155404 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.155341    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wg6m\" (UniqueName: \"kubernetes.io/projected/aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df-kube-api-access-4wg6m\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:03.155404 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.155354    5061 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df-workload-certs\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:03.155404 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.155368    5061 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf-credential-socket\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:03.155404 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.155381    5061 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03533dfe-8974-487d-9af6-4adda6fc8d8b-istio-envoy\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:03.155404 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.155394    5061 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df-istio-envoy\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:03.155404 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.155409    5061 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03533dfe-8974-487d-9af6-4adda6fc8d8b-istio-data\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:03.156351 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.155422    5061 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d238b910-dd97-4901-8e16-274b063b66ec-istio-podinfo\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:03.156351 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.155434    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ht4x\" (UniqueName: \"kubernetes.io/projected/3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf-kube-api-access-7ht4x\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:03.156351 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.155446    5061 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df-istio-token\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:03.156351 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.155458    5061 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d238b910-dd97-4901-8e16-274b063b66ec-istio-data\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:03.156351 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.155471    5061 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03533dfe-8974-487d-9af6-4adda6fc8d8b-credential-socket\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:03.156351 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.155482    5061 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf-istio-podinfo\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:03.156351 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.155494    5061 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03533dfe-8974-487d-9af6-4adda6fc8d8b-workload-certs\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:03.156351 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.155506    5061 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d238b910-dd97-4901-8e16-274b063b66ec-istio-envoy\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:03.501309 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.501224    5061 generic.go:356] "Generic (PLEG): container finished" podID="3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf" containerID="bb5f3d13f979cbf2216cd47e9e0c6af992a4f3cda415ac5fd12663e716e9d48c" exitCode=0
Oct 30 22:11:03.501309 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.501269    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5zvwr"
Oct 30 22:11:03.501864 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.501308    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5zvwr" event={"ID":"3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf","Type":"ContainerDied","Data":"bb5f3d13f979cbf2216cd47e9e0c6af992a4f3cda415ac5fd12663e716e9d48c"}
Oct 30 22:11:03.501864 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.501372    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5zvwr" event={"ID":"3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf","Type":"ContainerDied","Data":"df7501a516cfcb486cb7e48b31a0b42c1a35d9a7520fd78f0ec01a2cf652f43d"}
Oct 30 22:11:03.501864 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.501405    5061 scope.go:122] "RemoveContainer" containerID="bb5f3d13f979cbf2216cd47e9e0c6af992a4f3cda415ac5fd12663e716e9d48c"
Oct 30 22:11:03.502954 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.502938    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-kmtkh"
Oct 30 22:11:03.503071 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.502949    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-kmtkh" event={"ID":"03533dfe-8974-487d-9af6-4adda6fc8d8b","Type":"ContainerDied","Data":"5294fed5816bfc7761f8b107540477c1ee2bdae97896cac1f8d9ca308a2e9a1e"}
Oct 30 22:11:03.505610 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.505591    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-kdmlq"
Oct 30 22:11:03.505716 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.505584    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-kdmlq" event={"ID":"1aec235a-1d74-4044-aa3d-bd87a6a2d86f","Type":"ContainerDied","Data":"bf81e773ad5755d83b1aee2d238d9e855a27a44e799b994b6cdf66edfc3bfb54"}
Oct 30 22:11:03.507243 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.507151    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-4j4xr"
Oct 30 22:11:03.507243 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.507195    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-4j4xr" event={"ID":"aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df","Type":"ContainerDied","Data":"8b31f107fa355aad889b32b9f53a79d5f7647f23618b5498c67cef86a38595e6"}
Oct 30 22:11:03.510918 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.510743    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-9kcgf" event={"ID":"9b521eca-5188-43b6-b589-d0ccab8c4520","Type":"ContainerDied","Data":"1172fe2fb0d1371137d5d596eaa1e54e194e58ab1a82629769b0fa44b913ba51"}
Oct 30 22:11:03.510918 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.510766    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-9kcgf"
Oct 30 22:11:03.514058 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.513033    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-rh8lc" event={"ID":"d238b910-dd97-4901-8e16-274b063b66ec","Type":"ContainerDied","Data":"887e4f269583fbf7e5e8c18034308fa557ac836cab26b11b7bee6c8f7470502c"}
Oct 30 22:11:03.514058 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.513057    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-rh8lc"
Oct 30 22:11:03.514571 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.514423    5061 scope.go:122] "RemoveContainer" containerID="bb5f3d13f979cbf2216cd47e9e0c6af992a4f3cda415ac5fd12663e716e9d48c"
Oct 30 22:11:03.514744 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:11:03.514716    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb5f3d13f979cbf2216cd47e9e0c6af992a4f3cda415ac5fd12663e716e9d48c\": container with ID starting with bb5f3d13f979cbf2216cd47e9e0c6af992a4f3cda415ac5fd12663e716e9d48c not found: ID does not exist" containerID="bb5f3d13f979cbf2216cd47e9e0c6af992a4f3cda415ac5fd12663e716e9d48c"
Oct 30 22:11:03.514826 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.514744    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb5f3d13f979cbf2216cd47e9e0c6af992a4f3cda415ac5fd12663e716e9d48c"} err="failed to get container status \"bb5f3d13f979cbf2216cd47e9e0c6af992a4f3cda415ac5fd12663e716e9d48c\": rpc error: code = NotFound desc = could not find container \"bb5f3d13f979cbf2216cd47e9e0c6af992a4f3cda415ac5fd12663e716e9d48c\": container with ID starting with bb5f3d13f979cbf2216cd47e9e0c6af992a4f3cda415ac5fd12663e716e9d48c not found: ID does not exist"
Oct 30 22:11:03.514826 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.514777    5061 scope.go:122] "RemoveContainer" containerID="49633a346981edd93f36051e1be89b70e4aed785827492ede902087003443483"
Oct 30 22:11:03.539060 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.539039    5061 scope.go:122] "RemoveContainer" containerID="30a4ed5a6cb2e9a2845117356449fa90af9c06c206524fcf5bf3976b75992727"
Oct 30 22:11:03.548928 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.548911    5061 scope.go:122] "RemoveContainer" containerID="8b977b2c9b1fc9c6ae43d67d2948b12ffc7573a3cf71688b8bc9de22f1bf0fea"
Oct 30 22:11:03.558137 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.558119    5061 scope.go:122] "RemoveContainer" containerID="bcb2e4028f11ad1fa672f8a687f6cbd4285f037f8856cfeb5e62b899b3f87113"
Oct 30 22:11:03.568918 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.568893    5061 scope.go:122] "RemoveContainer" containerID="a0d98cfa8080dbbca0afe6db33b277854ae57aea174343d33e4a310c664dc2aa"
Oct 30 22:11:03.658856 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.658823    5061 patch_prober.go:29] interesting pod/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-9kcgf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:11:03.659027 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.658897    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-9kcgf" podUID="9b521eca-5188-43b6-b589-d0ccab8c4520" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:03.684026 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.683989    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-rh8lc"]
Oct 30 22:11:03.691680 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.691653    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-rh8lc"]
Oct 30 22:11:03.805173 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.805139    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-kmtkh"]
Oct 30 22:11:03.812494 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.812468    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-kmtkh"]
Oct 30 22:11:03.971333 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.971302    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-kdmlq"]
Oct 30 22:11:03.989094 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:03.989062    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-kdmlq"]
Oct 30 22:11:04.105738 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:04.105646    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-4j4xr"]
Oct 30 22:11:04.111407 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:04.111379    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-4j4xr"]
Oct 30 22:11:04.227113 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:04.227076    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-9kcgf"]
Oct 30 22:11:04.232541 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:04.232509    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-9kcgf"]
Oct 30 22:11:04.347702 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:04.347672    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5zvwr"]
Oct 30 22:11:04.351168 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:04.351146    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5zvwr"]
Oct 30 22:11:04.530291 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:04.530216    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03533dfe-8974-487d-9af6-4adda6fc8d8b" path="/var/lib/kubelet/pods/03533dfe-8974-487d-9af6-4adda6fc8d8b/volumes"
Oct 30 22:11:04.530897 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:04.530878    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1aec235a-1d74-4044-aa3d-bd87a6a2d86f" path="/var/lib/kubelet/pods/1aec235a-1d74-4044-aa3d-bd87a6a2d86f/volumes"
Oct 30 22:11:04.531371 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:04.531353    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf" path="/var/lib/kubelet/pods/3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf/volumes"
Oct 30 22:11:04.532161 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:04.532140    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b521eca-5188-43b6-b589-d0ccab8c4520" path="/var/lib/kubelet/pods/9b521eca-5188-43b6-b589-d0ccab8c4520/volumes"
Oct 30 22:11:04.532620 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:04.532604    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df" path="/var/lib/kubelet/pods/aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df/volumes"
Oct 30 22:11:04.533055 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:04.533041    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d238b910-dd97-4901-8e16-274b063b66ec" path="/var/lib/kubelet/pods/d238b910-dd97-4901-8e16-274b063b66ec/volumes"
Oct 30 22:11:20.689499 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:20.689418    5061 generic.go:356] "Generic (PLEG): container finished" podID="ce46d672-35a3-4cca-a9fc-b070939de95f" containerID="614f26a23cbceea26931c4ad079465625aa89f7b37390644c6d0999bb51b7970" exitCode=0
Oct 30 22:11:20.689933 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:20.689490    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7d434591-openshift-default-58f74c4865-rm5js" event={"ID":"ce46d672-35a3-4cca-a9fc-b070939de95f","Type":"ContainerDied","Data":"614f26a23cbceea26931c4ad079465625aa89f7b37390644c6d0999bb51b7970"}
Oct 30 22:11:20.689976 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:20.689959    5061 scope.go:122] "RemoveContainer" containerID="614f26a23cbceea26931c4ad079465625aa89f7b37390644c6d0999bb51b7970"
Oct 30 22:11:20.691718 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:20.691687    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:11:20.691845 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:20.691757    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:11:20.691845 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:20.691792    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:11:21.705513 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:21.705460    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7d434591-openshift-default-58f74c4865-rm5js" event={"ID":"ce46d672-35a3-4cca-a9fc-b070939de95f","Type":"ContainerStarted","Data":"1ab601cd0d39dbde394f0282268e4927c3b47d76885539c00d042e65b53d13fc"}
Oct 30 22:11:21.832869 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:21.830924    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-7d434591-openshift-default-58f74c4865-rm5js"
Oct 30 22:11:21.832869 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:21.831078    5061 patch_prober.go:29] interesting pod/e2e-mt-7d434591-openshift-default-58f74c4865-rm5js container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Oct 30 22:11:21.832869 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:21.831122    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7d434591-openshift-default-58f74c4865-rm5js" podUID="ce46d672-35a3-4cca-a9fc-b070939de95f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Oct 30 22:11:22.825551 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:22.823973    5061 patch_prober.go:29] interesting pod/e2e-mt-7d434591-openshift-default-58f74c4865-rm5js container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Oct 30 22:11:22.825551 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:22.824040    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7d434591-openshift-default-58f74c4865-rm5js" podUID="ce46d672-35a3-4cca-a9fc-b070939de95f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Oct 30 22:11:23.824555 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:23.823946    5061 patch_prober.go:29] interesting pod/e2e-mt-7d434591-openshift-default-58f74c4865-rm5js container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Oct 30 22:11:23.824555 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:23.824027    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7d434591-openshift-default-58f74c4865-rm5js" podUID="ce46d672-35a3-4cca-a9fc-b070939de95f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Oct 30 22:11:24.671786 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:11:24.671746    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 22:11:24.833550 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:24.832789    5061 patch_prober.go:29] interesting pod/e2e-mt-7d434591-openshift-default-58f74c4865-rm5js container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Oct 30 22:11:24.833550 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:24.832851    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7d434591-openshift-default-58f74c4865-rm5js" podUID="ce46d672-35a3-4cca-a9fc-b070939de95f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Oct 30 22:11:25.832913 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:25.832875    5061 patch_prober.go:29] interesting pod/e2e-mt-7d434591-openshift-default-58f74c4865-rm5js container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Oct 30 22:11:25.833313 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:25.832942    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7d434591-openshift-default-58f74c4865-rm5js" podUID="ce46d672-35a3-4cca-a9fc-b070939de95f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Oct 30 22:11:26.828552 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:26.826893    5061 patch_prober.go:29] interesting pod/e2e-mt-7d434591-openshift-default-58f74c4865-rm5js container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Oct 30 22:11:26.828552 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:26.826953    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7d434591-openshift-default-58f74c4865-rm5js" podUID="ce46d672-35a3-4cca-a9fc-b070939de95f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Oct 30 22:11:27.077080 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:27.077044    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-7d434591-openshift-default-58f74c4865-rm5js"
Oct 30 22:11:27.829790 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:27.829760    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-7d434591-openshift-default-58f74c4865-rm5js"
Oct 30 22:11:27.830896 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:27.830872    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-7d434591-openshift-default-58f74c4865-rm5js"
Oct 30 22:11:28.201141 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:28.201055    5061 scope.go:122] "RemoveContainer" containerID="93ceb97a68e08e3dad5e14310f54634b405fdf465e0d5a27d43ffb00d18735ca"
Oct 30 22:11:28.214714 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:11:28.214687    5061 scope.go:122] "RemoveContainer" containerID="843e34dea9f18cdfa8d420a37f29e87ace6d8c307d1bca6247e223d1f3571283"
Oct 30 22:11:54.679016 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:11:54.678979    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 22:12:24.667737 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:12:24.667695    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 22:12:26.790006 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:12:26.789976    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-sclrt_5e74071e-0462-42b5-a05c-d90bb638478a/console-operator/2.log"
Oct 30 22:12:26.794542 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:12:26.794509    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-sclrt_5e74071e-0462-42b5-a05c-d90bb638478a/console-operator/2.log"
Oct 30 22:12:31.318336 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:12:31.318297    5061 generic.go:356] "Generic (PLEG): container finished" podID="83adee7e-3585-4d0d-b1b2-d4b5d3be0588" containerID="28d948b4308d1a2ef8191c64f52379ad5384f4efea11fb8817b380e34d76df2d" exitCode=0
Oct 30 22:12:31.318835 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:12:31.318379    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-9r5c8" event={"ID":"83adee7e-3585-4d0d-b1b2-d4b5d3be0588","Type":"ContainerDied","Data":"28d948b4308d1a2ef8191c64f52379ad5384f4efea11fb8817b380e34d76df2d"}
Oct 30 22:12:31.318835 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:12:31.318806    5061 scope.go:122] "RemoveContainer" containerID="28d948b4308d1a2ef8191c64f52379ad5384f4efea11fb8817b380e34d76df2d"
Oct 30 22:12:31.320472 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:12:31.320438    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:12:31.320625 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:12:31.320520    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:12:31.320625 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:12:31.320587    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:12:32.342420 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:12:32.342381    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-9r5c8" event={"ID":"83adee7e-3585-4d0d-b1b2-d4b5d3be0588","Type":"ContainerStarted","Data":"6a6cc21e10bc1e0353ea3b36fe7f807ea2a367e761da2d139b8e0f3b8009c1f0"}
Oct 30 22:12:32.774951 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:12:32.774864    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-9r5c8"
Oct 30 22:12:33.204056 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:12:33.204022    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-9r5c8"
Oct 30 22:12:33.205615 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:12:33.205591    5061 patch_prober.go:29] interesting pod/e2e-mt-778d39d3-openshift-default-bff74755c-9r5c8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body=
Oct 30 22:12:33.205734 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:12:33.205645    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-9r5c8" podUID="83adee7e-3585-4d0d-b1b2-d4b5d3be0588" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused"
Oct 30 22:12:34.203440 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:12:34.203407    5061 patch_prober.go:29] interesting pod/e2e-mt-778d39d3-openshift-default-bff74755c-9r5c8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body=
Oct 30 22:12:34.203844 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:12:34.203457    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-9r5c8" podUID="83adee7e-3585-4d0d-b1b2-d4b5d3be0588" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused"
Oct 30 22:12:35.203600 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:12:35.203568    5061 patch_prober.go:29] interesting pod/e2e-mt-778d39d3-openshift-default-bff74755c-9r5c8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body=
Oct 30 22:12:35.204004 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:12:35.203618    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-9r5c8" podUID="83adee7e-3585-4d0d-b1b2-d4b5d3be0588" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused"
Oct 30 22:12:36.203733 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:12:36.203691    5061 patch_prober.go:29] interesting pod/e2e-mt-778d39d3-openshift-default-bff74755c-9r5c8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body=
Oct 30 22:12:36.204145 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:12:36.203755    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-9r5c8" podUID="83adee7e-3585-4d0d-b1b2-d4b5d3be0588" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused"
Oct 30 22:12:37.203794 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:12:37.203760    5061 patch_prober.go:29] interesting pod/e2e-mt-778d39d3-openshift-default-bff74755c-9r5c8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body=
Oct 30 22:12:37.204289 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:12:37.203841    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-9r5c8" podUID="83adee7e-3585-4d0d-b1b2-d4b5d3be0588" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused"
Oct 30 22:12:38.208387 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:12:38.208351    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-9r5c8"
Oct 30 22:12:38.209471 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:12:38.209449    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-9r5c8"
Oct 30 22:12:48.870855 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:12:48.870814    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7d434591-bjs4d"]
Oct 30 22:12:48.883508 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:12:48.883471    5061 cpu_manager.go:414] "containerMap: removing container" podUID="d238b910-dd97-4901-8e16-274b063b66ec" containerName="istio-proxy"
Oct 30 22:12:48.884668 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:12:48.884646    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d238b910-dd97-4901-8e16-274b063b66ec" containerName="istio-proxy"
Oct 30 22:12:48.884848 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:12:48.884832    5061 cpu_manager.go:414] "containerMap: removing container" podUID="3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf" containerName="istio-proxy"
Oct 30 22:12:48.884941 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:12:48.884930    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf" containerName="istio-proxy"
Oct 30 22:12:48.885027 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:12:48.885017    5061 cpu_manager.go:414] "containerMap: removing container" podUID="1aec235a-1d74-4044-aa3d-bd87a6a2d86f" containerName="istio-proxy"
Oct 30 22:12:48.885109 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:12:48.885098    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1aec235a-1d74-4044-aa3d-bd87a6a2d86f" containerName="istio-proxy"
Oct 30 22:12:48.885216 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:12:48.885206    5061 cpu_manager.go:414] "containerMap: removing container" podUID="9b521eca-5188-43b6-b589-d0ccab8c4520" containerName="istio-proxy"
Oct 30 22:12:48.885291 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:12:48.885280    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b521eca-5188-43b6-b589-d0ccab8c4520" containerName="istio-proxy"
Oct 30 22:12:48.885398 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:12:48.885389    5061 cpu_manager.go:414] "containerMap: removing container" podUID="03533dfe-8974-487d-9af6-4adda6fc8d8b" containerName="istio-proxy"
Oct 30 22:12:48.885475 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:12:48.885464    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03533dfe-8974-487d-9af6-4adda6fc8d8b" containerName="istio-proxy"
Oct 30 22:12:48.885631 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:12:48.885618    5061 cpu_manager.go:414] "containerMap: removing container" podUID="aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df" containerName="istio-proxy"
Oct 30 22:12:48.885719 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:12:48.885708    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df" containerName="istio-proxy"
Oct 30 22:12:48.891961 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:12:48.891935    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b521eca-5188-43b6-b589-d0ccab8c4520" containerName="istio-proxy"
Oct 30 22:12:48.892136 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:12:48.892122    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="03533dfe-8974-487d-9af6-4adda6fc8d8b" containerName="istio-proxy"
Oct 30 22:12:48.892257 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:12:48.892246    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa7b6c04-24c2-40b5-8d2a-d0c4da6f41df" containerName="istio-proxy"
Oct 30 22:12:48.892344 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:12:48.892334    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="d238b910-dd97-4901-8e16-274b063b66ec" containerName="istio-proxy"
Oct 30 22:12:48.892427 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:12:48.892418    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e4c8b0f-0e9a-4dd3-b405-f9c669d3ccaf" containerName="istio-proxy"
Oct 30 22:12:48.892506 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:12:48.892496    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="1aec235a-1d74-4044-aa3d-bd87a6a2d86f" containerName="istio-proxy"
Oct 30 22:12:48.902650 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:12:48.902623    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7d434591-bjs4d"
Oct 30 22:12:48.929099 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:12:48.929028    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7d434591-bjs4d"]
Oct 30 22:12:48.968591 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:12:48.963878    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gdmfg\""
Oct 30 22:12:48.968591 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:12:48.964518    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:12:48.976609 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:12:48.969827    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/202217c5-cd5c-4abd-8081-342d904e86f1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-7d434591-bjs4d\" (UID: \"202217c5-cd5c-4abd-8081-342d904e86f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7d434591-bjs4d"
Oct 30 22:12:48.976609 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:12:48.969887    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/202217c5-cd5c-4abd-8081-342d904e86f1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-7d434591-bjs4d\" (UID: \"202217c5-cd5c-4abd-8081-342d904e86f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7d434591-bjs4d"
Oct 30 22:12:48.976609 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:12:48.969936    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtcl6\" (UniqueName: \"kubernetes.io/projected/202217c5-cd5c-4abd-8081-342d904e86f1-kube-api-access-jtcl6\") pod \"maas-api-revoke-keys-e2e-mt-7d434591-bjs4d\" (UID: \"202217c5-cd5c-4abd-8081-342d904e86f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7d434591-bjs4d"
Oct 30 22:12:49.071839 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:12:49.071793    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/202217c5-cd5c-4abd-8081-342d904e86f1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-7d434591-bjs4d\" (UID: \"202217c5-cd5c-4abd-8081-342d904e86f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7d434591-bjs4d"
Oct 30 22:12:49.072093 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:12:49.071867    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/202217c5-cd5c-4abd-8081-342d904e86f1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-7d434591-bjs4d\" (UID: \"202217c5-cd5c-4abd-8081-342d904e86f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7d434591-bjs4d"
Oct 30 22:12:49.072093 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:12:49.071904    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtcl6\" (UniqueName: \"kubernetes.io/projected/202217c5-cd5c-4abd-8081-342d904e86f1-kube-api-access-jtcl6\") pod \"maas-api-revoke-keys-e2e-mt-7d434591-bjs4d\" (UID: \"202217c5-cd5c-4abd-8081-342d904e86f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7d434591-bjs4d"
Oct 30 22:12:49.072726 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:12:49.072699    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/202217c5-cd5c-4abd-8081-342d904e86f1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-7d434591-bjs4d\" (UID: \"202217c5-cd5c-4abd-8081-342d904e86f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7d434591-bjs4d"
Oct 30 22:12:49.074373 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:12:49.074343    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/202217c5-cd5c-4abd-8081-342d904e86f1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-7d434591-bjs4d\" (UID: \"202217c5-cd5c-4abd-8081-342d904e86f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7d434591-bjs4d"
Oct 30 22:12:49.134021 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:12:49.133940    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtcl6\" (UniqueName: \"kubernetes.io/projected/202217c5-cd5c-4abd-8081-342d904e86f1-kube-api-access-jtcl6\") pod \"maas-api-revoke-keys-e2e-mt-7d434591-bjs4d\" (UID: \"202217c5-cd5c-4abd-8081-342d904e86f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7d434591-bjs4d"
Oct 30 22:12:49.299620 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:12:49.299582    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7d434591-bjs4d"
Oct 30 22:12:49.898115 ip-10-0-130-60 kubenswrapper[5061]: W1030 22:12:49.898064    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod202217c5_cd5c_4abd_8081_342d904e86f1.slice/crio-feb2716f6e99e073076d4069972b617eaee6c5242ff7b7681fcb2ce40cbfb01c WatchSource:0}: Error finding container feb2716f6e99e073076d4069972b617eaee6c5242ff7b7681fcb2ce40cbfb01c: Status 404 returned error can't find the container with id feb2716f6e99e073076d4069972b617eaee6c5242ff7b7681fcb2ce40cbfb01c
Oct 30 22:12:49.898605 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:12:49.898431    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7d434591-bjs4d"]
Oct 30 22:12:50.550937 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:12:50.550897    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7d434591-bjs4d" event={"ID":"202217c5-cd5c-4abd-8081-342d904e86f1","Type":"ContainerStarted","Data":"3b52d61ffc4eb7b8d1a162566939e46bf8e201a1d92bbe5360b854d6e1080245"}
Oct 30 22:12:50.551112 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:12:50.550953    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7d434591-bjs4d" event={"ID":"202217c5-cd5c-4abd-8081-342d904e86f1","Type":"ContainerStarted","Data":"feb2716f6e99e073076d4069972b617eaee6c5242ff7b7681fcb2ce40cbfb01c"}
Oct 30 22:12:50.666978 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:12:50.666880    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7d434591-bjs4d" podStartSLOduration=2.666865419 podStartE2EDuration="2.666865419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:50.66249209 +0000 UTC m=+1824.275261848" watchObservedRunningTime="2026-10-30 22:12:50.666865419 +0000 UTC m=+1824.279635176"
Oct 30 22:12:51.558976 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:12:51.558940    5061 generic.go:356] "Generic (PLEG): container finished" podID="202217c5-cd5c-4abd-8081-342d904e86f1" containerID="3b52d61ffc4eb7b8d1a162566939e46bf8e201a1d92bbe5360b854d6e1080245" exitCode=0
Oct 30 22:12:51.559410 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:12:51.559017    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7d434591-bjs4d" event={"ID":"202217c5-cd5c-4abd-8081-342d904e86f1","Type":"ContainerDied","Data":"3b52d61ffc4eb7b8d1a162566939e46bf8e201a1d92bbe5360b854d6e1080245"}
Oct 30 22:12:52.734920 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:12:52.734895    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7d434591-bjs4d"
Oct 30 22:12:52.855863 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:12:52.855790    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/202217c5-cd5c-4abd-8081-342d904e86f1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/202217c5-cd5c-4abd-8081-342d904e86f1-maas-api-openshift-ca\") pod \"202217c5-cd5c-4abd-8081-342d904e86f1\" (UID: \"202217c5-cd5c-4abd-8081-342d904e86f1\") "
Oct 30 22:12:52.855863 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:12:52.855825    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/202217c5-cd5c-4abd-8081-342d904e86f1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/202217c5-cd5c-4abd-8081-342d904e86f1-maas-api-xks-ca\") pod \"202217c5-cd5c-4abd-8081-342d904e86f1\" (UID: \"202217c5-cd5c-4abd-8081-342d904e86f1\") "
Oct 30 22:12:52.855863 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:12:52.855845    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/202217c5-cd5c-4abd-8081-342d904e86f1-kube-api-access-jtcl6\" (UniqueName: \"kubernetes.io/projected/202217c5-cd5c-4abd-8081-342d904e86f1-kube-api-access-jtcl6\") pod \"202217c5-cd5c-4abd-8081-342d904e86f1\" (UID: \"202217c5-cd5c-4abd-8081-342d904e86f1\") "
Oct 30 22:12:52.856230 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:12:52.856207    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/202217c5-cd5c-4abd-8081-342d904e86f1-maas-api-openshift-ca" pod "202217c5-cd5c-4abd-8081-342d904e86f1" (UID: "202217c5-cd5c-4abd-8081-342d904e86f1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:12:52.856509 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:12:52.856491    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/202217c5-cd5c-4abd-8081-342d904e86f1-maas-api-openshift-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:12:52.857977 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:12:52.857956    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/202217c5-cd5c-4abd-8081-342d904e86f1-maas-api-xks-ca" pod "202217c5-cd5c-4abd-8081-342d904e86f1" (UID: "202217c5-cd5c-4abd-8081-342d904e86f1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:12:52.858110 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:12:52.858096    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/202217c5-cd5c-4abd-8081-342d904e86f1-kube-api-access-jtcl6" pod "202217c5-cd5c-4abd-8081-342d904e86f1" (UID: "202217c5-cd5c-4abd-8081-342d904e86f1"). InnerVolumeSpecName "kube-api-access-jtcl6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:12:52.958327 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:12:52.958287    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/202217c5-cd5c-4abd-8081-342d904e86f1-maas-api-xks-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:12:52.958479 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:12:52.958330    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtcl6\" (UniqueName: \"kubernetes.io/projected/202217c5-cd5c-4abd-8081-342d904e86f1-kube-api-access-jtcl6\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:12:53.577123 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:12:53.577091    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7d434591-bjs4d"
Oct 30 22:12:53.577303 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:12:53.577133    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7d434591-bjs4d" event={"ID":"202217c5-cd5c-4abd-8081-342d904e86f1","Type":"ContainerDied","Data":"feb2716f6e99e073076d4069972b617eaee6c5242ff7b7681fcb2ce40cbfb01c"}
Oct 30 22:12:53.577303 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:12:53.577177    5061 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="feb2716f6e99e073076d4069972b617eaee6c5242ff7b7681fcb2ce40cbfb01c"
Oct 30 22:12:54.671444 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:12:54.671412    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 22:12:57.277854 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:12:57.277819    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-m4q9b"]
Oct 30 22:12:57.278328 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:12:57.278036    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-m4q9b" podUID="fd9cb1c8-e190-4652-b11d-81299bbd7bcb" containerName="istio-proxy" containerID="cri-o://9ac966264097d395670ddfdde2337c93ce57769db01bed0c7e5ff4ae9f502d07" gracePeriod=30
Oct 30 22:13:02.574055 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:02.574023    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-m4q9b"
Oct 30 22:13:02.644097 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:02.644055    5061 generic.go:356] "Generic (PLEG): container finished" podID="fd9cb1c8-e190-4652-b11d-81299bbd7bcb" containerID="9ac966264097d395670ddfdde2337c93ce57769db01bed0c7e5ff4ae9f502d07" exitCode=0
Oct 30 22:13:02.644269 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:02.644103    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-m4q9b"
Oct 30 22:13:02.644269 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:02.644146    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-m4q9b" event={"ID":"fd9cb1c8-e190-4652-b11d-81299bbd7bcb","Type":"ContainerDied","Data":"9ac966264097d395670ddfdde2337c93ce57769db01bed0c7e5ff4ae9f502d07"}
Oct 30 22:13:02.644269 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:02.644211    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-m4q9b" event={"ID":"fd9cb1c8-e190-4652-b11d-81299bbd7bcb","Type":"ContainerDied","Data":"244e4504cc3374316f081bcc38cba4b74a3760a556d2f4a4406cdc2e28d6d55f"}
Oct 30 22:13:02.644269 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:02.644243    5061 scope.go:122] "RemoveContainer" containerID="9ac966264097d395670ddfdde2337c93ce57769db01bed0c7e5ff4ae9f502d07"
Oct 30 22:13:02.655198 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:02.655181    5061 scope.go:122] "RemoveContainer" containerID="9ac966264097d395670ddfdde2337c93ce57769db01bed0c7e5ff4ae9f502d07"
Oct 30 22:13:02.655466 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:13:02.655446    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ac966264097d395670ddfdde2337c93ce57769db01bed0c7e5ff4ae9f502d07\": container with ID starting with 9ac966264097d395670ddfdde2337c93ce57769db01bed0c7e5ff4ae9f502d07 not found: ID does not exist" containerID="9ac966264097d395670ddfdde2337c93ce57769db01bed0c7e5ff4ae9f502d07"
Oct 30 22:13:02.655523 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:02.655474    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ac966264097d395670ddfdde2337c93ce57769db01bed0c7e5ff4ae9f502d07"} err="failed to get container status \"9ac966264097d395670ddfdde2337c93ce57769db01bed0c7e5ff4ae9f502d07\": rpc error: code = NotFound desc = could not find container \"9ac966264097d395670ddfdde2337c93ce57769db01bed0c7e5ff4ae9f502d07\": container with ID starting with 9ac966264097d395670ddfdde2337c93ce57769db01bed0c7e5ff4ae9f502d07 not found: ID does not exist"
Oct 30 22:13:02.699858 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:02.699836    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7d434591-6f89ff7d59-kbg9t"]
Oct 30 22:13:02.720143 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:02.720116    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd9cb1c8-e190-4652-b11d-81299bbd7bcb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd9cb1c8-e190-4652-b11d-81299bbd7bcb-istio-envoy\") pod \"fd9cb1c8-e190-4652-b11d-81299bbd7bcb\" (UID: \"fd9cb1c8-e190-4652-b11d-81299bbd7bcb\") "
Oct 30 22:13:02.720274 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:02.720162    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd9cb1c8-e190-4652-b11d-81299bbd7bcb-istio-token\" (UniqueName: \"kubernetes.io/projected/fd9cb1c8-e190-4652-b11d-81299bbd7bcb-istio-token\") pod \"fd9cb1c8-e190-4652-b11d-81299bbd7bcb\" (UID: \"fd9cb1c8-e190-4652-b11d-81299bbd7bcb\") "
Oct 30 22:13:02.720274 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:02.720234    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fd9cb1c8-e190-4652-b11d-81299bbd7bcb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd9cb1c8-e190-4652-b11d-81299bbd7bcb-istio-podinfo\") pod \"fd9cb1c8-e190-4652-b11d-81299bbd7bcb\" (UID: \"fd9cb1c8-e190-4652-b11d-81299bbd7bcb\") "
Oct 30 22:13:02.720406 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:02.720271    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd9cb1c8-e190-4652-b11d-81299bbd7bcb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd9cb1c8-e190-4652-b11d-81299bbd7bcb-credential-socket\") pod \"fd9cb1c8-e190-4652-b11d-81299bbd7bcb\" (UID: \"fd9cb1c8-e190-4652-b11d-81299bbd7bcb\") "
Oct 30 22:13:02.720512 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:02.720492    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd9cb1c8-e190-4652-b11d-81299bbd7bcb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd9cb1c8-e190-4652-b11d-81299bbd7bcb-istiod-ca-cert\") pod \"fd9cb1c8-e190-4652-b11d-81299bbd7bcb\" (UID: \"fd9cb1c8-e190-4652-b11d-81299bbd7bcb\") "
Oct 30 22:13:02.720776 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:02.720753    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd9cb1c8-e190-4652-b11d-81299bbd7bcb-credential-socket" pod "fd9cb1c8-e190-4652-b11d-81299bbd7bcb" (UID: "fd9cb1c8-e190-4652-b11d-81299bbd7bcb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:13:02.720879 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:02.720829    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd9cb1c8-e190-4652-b11d-81299bbd7bcb-istiod-ca-cert" pod "fd9cb1c8-e190-4652-b11d-81299bbd7bcb" (UID: "fd9cb1c8-e190-4652-b11d-81299bbd7bcb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:13:02.720981 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:02.720961    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd9cb1c8-e190-4652-b11d-81299bbd7bcb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd9cb1c8-e190-4652-b11d-81299bbd7bcb-istio-data\") pod \"fd9cb1c8-e190-4652-b11d-81299bbd7bcb\" (UID: \"fd9cb1c8-e190-4652-b11d-81299bbd7bcb\") "
Oct 30 22:13:02.721076 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:02.721056    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd9cb1c8-e190-4652-b11d-81299bbd7bcb-istio-data" pod "fd9cb1c8-e190-4652-b11d-81299bbd7bcb" (UID: "fd9cb1c8-e190-4652-b11d-81299bbd7bcb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:13:02.721186 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:02.721170    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd9cb1c8-e190-4652-b11d-81299bbd7bcb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd9cb1c8-e190-4652-b11d-81299bbd7bcb-workload-socket\") pod \"fd9cb1c8-e190-4652-b11d-81299bbd7bcb\" (UID: \"fd9cb1c8-e190-4652-b11d-81299bbd7bcb\") "
Oct 30 22:13:02.721577 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:02.721557    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd9cb1c8-e190-4652-b11d-81299bbd7bcb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd9cb1c8-e190-4652-b11d-81299bbd7bcb-workload-certs\") pod \"fd9cb1c8-e190-4652-b11d-81299bbd7bcb\" (UID: \"fd9cb1c8-e190-4652-b11d-81299bbd7bcb\") "
Oct 30 22:13:02.721799 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:02.721460    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd9cb1c8-e190-4652-b11d-81299bbd7bcb-workload-socket" pod "fd9cb1c8-e190-4652-b11d-81299bbd7bcb" (UID: "fd9cb1c8-e190-4652-b11d-81299bbd7bcb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:13:02.721879 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:02.721765    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd9cb1c8-e190-4652-b11d-81299bbd7bcb-workload-certs" pod "fd9cb1c8-e190-4652-b11d-81299bbd7bcb" (UID: "fd9cb1c8-e190-4652-b11d-81299bbd7bcb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:13:02.722011 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:02.721984    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd9cb1c8-e190-4652-b11d-81299bbd7bcb-kube-api-access-kt2pk\" (UniqueName: \"kubernetes.io/projected/fd9cb1c8-e190-4652-b11d-81299bbd7bcb-kube-api-access-kt2pk\") pod \"fd9cb1c8-e190-4652-b11d-81299bbd7bcb\" (UID: \"fd9cb1c8-e190-4652-b11d-81299bbd7bcb\") "
Oct 30 22:13:02.722793 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:02.722768    5061 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd9cb1c8-e190-4652-b11d-81299bbd7bcb-istiod-ca-cert\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:13:02.722900 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:02.722798    5061 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd9cb1c8-e190-4652-b11d-81299bbd7bcb-istio-data\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:13:02.722900 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:02.722812    5061 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd9cb1c8-e190-4652-b11d-81299bbd7bcb-workload-socket\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:13:02.722900 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:02.722825    5061 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd9cb1c8-e190-4652-b11d-81299bbd7bcb-workload-certs\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:13:02.722900 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:02.722838    5061 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd9cb1c8-e190-4652-b11d-81299bbd7bcb-credential-socket\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:13:02.723143 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:02.723121    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd9cb1c8-e190-4652-b11d-81299bbd7bcb-istio-token" pod "fd9cb1c8-e190-4652-b11d-81299bbd7bcb" (UID: "fd9cb1c8-e190-4652-b11d-81299bbd7bcb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:13:02.723225 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:02.723151    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd9cb1c8-e190-4652-b11d-81299bbd7bcb-istio-envoy" pod "fd9cb1c8-e190-4652-b11d-81299bbd7bcb" (UID: "fd9cb1c8-e190-4652-b11d-81299bbd7bcb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:13:02.723929 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:02.723909    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fd9cb1c8-e190-4652-b11d-81299bbd7bcb-istio-podinfo" pod "fd9cb1c8-e190-4652-b11d-81299bbd7bcb" (UID: "fd9cb1c8-e190-4652-b11d-81299bbd7bcb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:13:02.724932 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:02.724907    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd9cb1c8-e190-4652-b11d-81299bbd7bcb-kube-api-access-kt2pk" pod "fd9cb1c8-e190-4652-b11d-81299bbd7bcb" (UID: "fd9cb1c8-e190-4652-b11d-81299bbd7bcb"). InnerVolumeSpecName "kube-api-access-kt2pk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:13:02.824361 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:02.824271    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kt2pk\" (UniqueName: \"kubernetes.io/projected/fd9cb1c8-e190-4652-b11d-81299bbd7bcb-kube-api-access-kt2pk\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:13:02.824361 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:02.824307    5061 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd9cb1c8-e190-4652-b11d-81299bbd7bcb-istio-envoy\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:13:02.824361 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:02.824321    5061 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd9cb1c8-e190-4652-b11d-81299bbd7bcb-istio-token\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:13:02.824361 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:02.824334    5061 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd9cb1c8-e190-4652-b11d-81299bbd7bcb-istio-podinfo\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:13:03.150092 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:03.150006    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-m4q9b"]
Oct 30 22:13:03.155510 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:03.155476    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-m4q9b"]
Oct 30 22:13:04.530373 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:04.530341    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd9cb1c8-e190-4652-b11d-81299bbd7bcb" path="/var/lib/kubelet/pods/fd9cb1c8-e190-4652-b11d-81299bbd7bcb/volumes"
Oct 30 22:13:07.700217 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:07.700175    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-7d434591-6f89ff7d59-kbg9t" podUID="6e43c8db-84a2-453e-8bac-04f149ddd7a6" containerName="payload-pre-processing" containerID="cri-o://b5aee796a5c954dbed27815703e412db7c6d3e0911e5b53536e663192732a29f" gracePeriod=25
Oct 30 22:13:08.302614 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:08.302584    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7d434591-6f89ff7d59-kbg9t"
Oct 30 22:13:08.398193 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:08.398161    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e43c8db-84a2-453e-8bac-04f149ddd7a6-kube-api-access-kkmw7\" (UniqueName: \"kubernetes.io/projected/6e43c8db-84a2-453e-8bac-04f149ddd7a6-kube-api-access-kkmw7\") pod \"6e43c8db-84a2-453e-8bac-04f149ddd7a6\" (UID: \"6e43c8db-84a2-453e-8bac-04f149ddd7a6\") "
Oct 30 22:13:08.398401 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:08.398272    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e43c8db-84a2-453e-8bac-04f149ddd7a6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e43c8db-84a2-453e-8bac-04f149ddd7a6-plugins-config-volume\") pod \"6e43c8db-84a2-453e-8bac-04f149ddd7a6\" (UID: \"6e43c8db-84a2-453e-8bac-04f149ddd7a6\") "
Oct 30 22:13:08.398735 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:08.398703    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e43c8db-84a2-453e-8bac-04f149ddd7a6-plugins-config-volume" pod "6e43c8db-84a2-453e-8bac-04f149ddd7a6" (UID: "6e43c8db-84a2-453e-8bac-04f149ddd7a6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:13:08.400313 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:08.400295    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e43c8db-84a2-453e-8bac-04f149ddd7a6-kube-api-access-kkmw7" pod "6e43c8db-84a2-453e-8bac-04f149ddd7a6" (UID: "6e43c8db-84a2-453e-8bac-04f149ddd7a6"). InnerVolumeSpecName "kube-api-access-kkmw7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:13:08.500565 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:08.500453    5061 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e43c8db-84a2-453e-8bac-04f149ddd7a6-plugins-config-volume\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:13:08.500565 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:08.500485    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkmw7\" (UniqueName: \"kubernetes.io/projected/6e43c8db-84a2-453e-8bac-04f149ddd7a6-kube-api-access-kkmw7\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:13:08.757852 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:08.757759    5061 generic.go:356] "Generic (PLEG): container finished" podID="6e43c8db-84a2-453e-8bac-04f149ddd7a6" containerID="b5aee796a5c954dbed27815703e412db7c6d3e0911e5b53536e663192732a29f" exitCode=0
Oct 30 22:13:08.757852 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:08.757820    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7d434591-6f89ff7d59-kbg9t"
Oct 30 22:13:08.758290 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:08.757851    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7d434591-6f89ff7d59-kbg9t" event={"ID":"6e43c8db-84a2-453e-8bac-04f149ddd7a6","Type":"ContainerDied","Data":"b5aee796a5c954dbed27815703e412db7c6d3e0911e5b53536e663192732a29f"}
Oct 30 22:13:08.758290 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:08.757904    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7d434591-6f89ff7d59-kbg9t" event={"ID":"6e43c8db-84a2-453e-8bac-04f149ddd7a6","Type":"ContainerDied","Data":"fc60c58c87f483cd089540ec8cd6a8ee2038bbe748d13f8ab26e851d52dccfd1"}
Oct 30 22:13:08.758290 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:08.757925    5061 scope.go:122] "RemoveContainer" containerID="b5aee796a5c954dbed27815703e412db7c6d3e0911e5b53536e663192732a29f"
Oct 30 22:13:08.768536 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:08.768501    5061 scope.go:122] "RemoveContainer" containerID="b5aee796a5c954dbed27815703e412db7c6d3e0911e5b53536e663192732a29f"
Oct 30 22:13:08.768825 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:13:08.768795    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5aee796a5c954dbed27815703e412db7c6d3e0911e5b53536e663192732a29f\": container with ID starting with b5aee796a5c954dbed27815703e412db7c6d3e0911e5b53536e663192732a29f not found: ID does not exist" containerID="b5aee796a5c954dbed27815703e412db7c6d3e0911e5b53536e663192732a29f"
Oct 30 22:13:08.768893 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:08.768832    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5aee796a5c954dbed27815703e412db7c6d3e0911e5b53536e663192732a29f"} err="failed to get container status \"b5aee796a5c954dbed27815703e412db7c6d3e0911e5b53536e663192732a29f\": rpc error: code = NotFound desc = could not find container \"b5aee796a5c954dbed27815703e412db7c6d3e0911e5b53536e663192732a29f\": container with ID starting with b5aee796a5c954dbed27815703e412db7c6d3e0911e5b53536e663192732a29f not found: ID does not exist"
Oct 30 22:13:08.915309 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:08.915279    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7d434591-6f89ff7d59-kbg9t"]
Oct 30 22:13:08.964419 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:08.964379    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7d434591-6f89ff7d59-kbg9t"]
Oct 30 22:13:10.530848 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:10.530814    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e43c8db-84a2-453e-8bac-04f149ddd7a6" path="/var/lib/kubelet/pods/6e43c8db-84a2-453e-8bac-04f149ddd7a6/volumes"
Oct 30 22:13:11.778723 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:11.778678    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-25de17-openshift-default-8687bbf6cc-p9mf7"]
Oct 30 22:13:11.779826 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:11.779798    5061 cpu_manager.go:414] "containerMap: removing container" podUID="202217c5-cd5c-4abd-8081-342d904e86f1" containerName="revoke-keys"
Oct 30 22:13:11.779826 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:11.779824    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="202217c5-cd5c-4abd-8081-342d904e86f1" containerName="revoke-keys"
Oct 30 22:13:11.779990 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:11.779888    5061 cpu_manager.go:414] "containerMap: removing container" podUID="fd9cb1c8-e190-4652-b11d-81299bbd7bcb" containerName="istio-proxy"
Oct 30 22:13:11.779990 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:11.779896    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd9cb1c8-e190-4652-b11d-81299bbd7bcb" containerName="istio-proxy"
Oct 30 22:13:11.779990 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:11.779932    5061 cpu_manager.go:414] "containerMap: removing container" podUID="6e43c8db-84a2-453e-8bac-04f149ddd7a6" containerName="payload-pre-processing"
Oct 30 22:13:11.779990 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:11.779941    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e43c8db-84a2-453e-8bac-04f149ddd7a6" containerName="payload-pre-processing"
Oct 30 22:13:11.780120 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:11.780080    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd9cb1c8-e190-4652-b11d-81299bbd7bcb" containerName="istio-proxy"
Oct 30 22:13:11.780120 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:11.780097    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e43c8db-84a2-453e-8bac-04f149ddd7a6" containerName="payload-pre-processing"
Oct 30 22:13:11.780120 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:11.780111    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="202217c5-cd5c-4abd-8081-342d904e86f1" containerName="revoke-keys"
Oct 30 22:13:11.785460 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:11.785441    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-25de17-openshift-default-8687bbf6cc-p9mf7"
Oct 30 22:13:11.844550 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:11.838673    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-25de17-openshift-default-dockercfg-78xzx\""
Oct 30 22:13:11.894604 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:11.884977    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f464e173-8ada-4978-967f-7897d5bc5823-istiod-ca-cert\") pod \"e2e-api-a-25de17-openshift-default-8687bbf6cc-p9mf7\" (UID: \"f464e173-8ada-4978-967f-7897d5bc5823\") " pod="openshift-ingress/e2e-api-a-25de17-openshift-default-8687bbf6cc-p9mf7"
Oct 30 22:13:11.894604 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:11.885049    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f464e173-8ada-4978-967f-7897d5bc5823-istio-data\") pod \"e2e-api-a-25de17-openshift-default-8687bbf6cc-p9mf7\" (UID: \"f464e173-8ada-4978-967f-7897d5bc5823\") " pod="openshift-ingress/e2e-api-a-25de17-openshift-default-8687bbf6cc-p9mf7"
Oct 30 22:13:11.894604 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:11.885118    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f464e173-8ada-4978-967f-7897d5bc5823-credential-socket\") pod \"e2e-api-a-25de17-openshift-default-8687bbf6cc-p9mf7\" (UID: \"f464e173-8ada-4978-967f-7897d5bc5823\") " pod="openshift-ingress/e2e-api-a-25de17-openshift-default-8687bbf6cc-p9mf7"
Oct 30 22:13:11.894604 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:11.885163    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f464e173-8ada-4978-967f-7897d5bc5823-workload-certs\") pod \"e2e-api-a-25de17-openshift-default-8687bbf6cc-p9mf7\" (UID: \"f464e173-8ada-4978-967f-7897d5bc5823\") " pod="openshift-ingress/e2e-api-a-25de17-openshift-default-8687bbf6cc-p9mf7"
Oct 30 22:13:11.894604 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:11.885186    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f464e173-8ada-4978-967f-7897d5bc5823-istio-podinfo\") pod \"e2e-api-a-25de17-openshift-default-8687bbf6cc-p9mf7\" (UID: \"f464e173-8ada-4978-967f-7897d5bc5823\") " pod="openshift-ingress/e2e-api-a-25de17-openshift-default-8687bbf6cc-p9mf7"
Oct 30 22:13:11.894604 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:11.885232    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f464e173-8ada-4978-967f-7897d5bc5823-istio-envoy\") pod \"e2e-api-a-25de17-openshift-default-8687bbf6cc-p9mf7\" (UID: \"f464e173-8ada-4978-967f-7897d5bc5823\") " pod="openshift-ingress/e2e-api-a-25de17-openshift-default-8687bbf6cc-p9mf7"
Oct 30 22:13:11.894604 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:11.885257    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cx2x4\" (UniqueName: \"kubernetes.io/projected/f464e173-8ada-4978-967f-7897d5bc5823-kube-api-access-cx2x4\") pod \"e2e-api-a-25de17-openshift-default-8687bbf6cc-p9mf7\" (UID: \"f464e173-8ada-4978-967f-7897d5bc5823\") " pod="openshift-ingress/e2e-api-a-25de17-openshift-default-8687bbf6cc-p9mf7"
Oct 30 22:13:11.894604 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:11.885299    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f464e173-8ada-4978-967f-7897d5bc5823-istio-token\") pod \"e2e-api-a-25de17-openshift-default-8687bbf6cc-p9mf7\" (UID: \"f464e173-8ada-4978-967f-7897d5bc5823\") " pod="openshift-ingress/e2e-api-a-25de17-openshift-default-8687bbf6cc-p9mf7"
Oct 30 22:13:11.894604 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:11.885337    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f464e173-8ada-4978-967f-7897d5bc5823-workload-socket\") pod \"e2e-api-a-25de17-openshift-default-8687bbf6cc-p9mf7\" (UID: \"f464e173-8ada-4978-967f-7897d5bc5823\") " pod="openshift-ingress/e2e-api-a-25de17-openshift-default-8687bbf6cc-p9mf7"
Oct 30 22:13:12.002836 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:11.999372    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f464e173-8ada-4978-967f-7897d5bc5823-istiod-ca-cert\") pod \"e2e-api-a-25de17-openshift-default-8687bbf6cc-p9mf7\" (UID: \"f464e173-8ada-4978-967f-7897d5bc5823\") " pod="openshift-ingress/e2e-api-a-25de17-openshift-default-8687bbf6cc-p9mf7"
Oct 30 22:13:12.002836 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:11.999445    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f464e173-8ada-4978-967f-7897d5bc5823-istio-data\") pod \"e2e-api-a-25de17-openshift-default-8687bbf6cc-p9mf7\" (UID: \"f464e173-8ada-4978-967f-7897d5bc5823\") " pod="openshift-ingress/e2e-api-a-25de17-openshift-default-8687bbf6cc-p9mf7"
Oct 30 22:13:12.002836 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:11.999512    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f464e173-8ada-4978-967f-7897d5bc5823-credential-socket\") pod \"e2e-api-a-25de17-openshift-default-8687bbf6cc-p9mf7\" (UID: \"f464e173-8ada-4978-967f-7897d5bc5823\") " pod="openshift-ingress/e2e-api-a-25de17-openshift-default-8687bbf6cc-p9mf7"
Oct 30 22:13:12.002836 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:11.999575    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f464e173-8ada-4978-967f-7897d5bc5823-workload-certs\") pod \"e2e-api-a-25de17-openshift-default-8687bbf6cc-p9mf7\" (UID: \"f464e173-8ada-4978-967f-7897d5bc5823\") " pod="openshift-ingress/e2e-api-a-25de17-openshift-default-8687bbf6cc-p9mf7"
Oct 30 22:13:12.002836 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:11.999599    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f464e173-8ada-4978-967f-7897d5bc5823-istio-podinfo\") pod \"e2e-api-a-25de17-openshift-default-8687bbf6cc-p9mf7\" (UID: \"f464e173-8ada-4978-967f-7897d5bc5823\") " pod="openshift-ingress/e2e-api-a-25de17-openshift-default-8687bbf6cc-p9mf7"
Oct 30 22:13:12.002836 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:11.999648    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f464e173-8ada-4978-967f-7897d5bc5823-istio-envoy\") pod \"e2e-api-a-25de17-openshift-default-8687bbf6cc-p9mf7\" (UID: \"f464e173-8ada-4978-967f-7897d5bc5823\") " pod="openshift-ingress/e2e-api-a-25de17-openshift-default-8687bbf6cc-p9mf7"
Oct 30 22:13:12.002836 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:11.999672    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cx2x4\" (UniqueName: \"kubernetes.io/projected/f464e173-8ada-4978-967f-7897d5bc5823-kube-api-access-cx2x4\") pod \"e2e-api-a-25de17-openshift-default-8687bbf6cc-p9mf7\" (UID: \"f464e173-8ada-4978-967f-7897d5bc5823\") " pod="openshift-ingress/e2e-api-a-25de17-openshift-default-8687bbf6cc-p9mf7"
Oct 30 22:13:12.002836 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:11.999713    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f464e173-8ada-4978-967f-7897d5bc5823-istio-token\") pod \"e2e-api-a-25de17-openshift-default-8687bbf6cc-p9mf7\" (UID: \"f464e173-8ada-4978-967f-7897d5bc5823\") " pod="openshift-ingress/e2e-api-a-25de17-openshift-default-8687bbf6cc-p9mf7"
Oct 30 22:13:12.002836 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:11.999749    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f464e173-8ada-4978-967f-7897d5bc5823-workload-socket\") pod \"e2e-api-a-25de17-openshift-default-8687bbf6cc-p9mf7\" (UID: \"f464e173-8ada-4978-967f-7897d5bc5823\") " pod="openshift-ingress/e2e-api-a-25de17-openshift-default-8687bbf6cc-p9mf7"
Oct 30 22:13:12.002836 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:12.000212    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f464e173-8ada-4978-967f-7897d5bc5823-workload-socket\") pod \"e2e-api-a-25de17-openshift-default-8687bbf6cc-p9mf7\" (UID: \"f464e173-8ada-4978-967f-7897d5bc5823\") " pod="openshift-ingress/e2e-api-a-25de17-openshift-default-8687bbf6cc-p9mf7"
Oct 30 22:13:12.010935 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:12.010084    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f464e173-8ada-4978-967f-7897d5bc5823-credential-socket\") pod \"e2e-api-a-25de17-openshift-default-8687bbf6cc-p9mf7\" (UID: \"f464e173-8ada-4978-967f-7897d5bc5823\") " pod="openshift-ingress/e2e-api-a-25de17-openshift-default-8687bbf6cc-p9mf7"
Oct 30 22:13:12.010935 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:12.010696    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f464e173-8ada-4978-967f-7897d5bc5823-istio-data\") pod \"e2e-api-a-25de17-openshift-default-8687bbf6cc-p9mf7\" (UID: \"f464e173-8ada-4978-967f-7897d5bc5823\") " pod="openshift-ingress/e2e-api-a-25de17-openshift-default-8687bbf6cc-p9mf7"
Oct 30 22:13:12.021325 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:12.011257    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f464e173-8ada-4978-967f-7897d5bc5823-workload-certs\") pod \"e2e-api-a-25de17-openshift-default-8687bbf6cc-p9mf7\" (UID: \"f464e173-8ada-4978-967f-7897d5bc5823\") " pod="openshift-ingress/e2e-api-a-25de17-openshift-default-8687bbf6cc-p9mf7"
Oct 30 22:13:12.029603 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:12.029448    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-25de17-openshift-default-8687bbf6cc-p9mf7"]
Oct 30 22:13:12.067552 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:12.060686    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f464e173-8ada-4978-967f-7897d5bc5823-istiod-ca-cert\") pod \"e2e-api-a-25de17-openshift-default-8687bbf6cc-p9mf7\" (UID: \"f464e173-8ada-4978-967f-7897d5bc5823\") " pod="openshift-ingress/e2e-api-a-25de17-openshift-default-8687bbf6cc-p9mf7"
Oct 30 22:13:12.071547 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:12.068821    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f464e173-8ada-4978-967f-7897d5bc5823-istio-envoy\") pod \"e2e-api-a-25de17-openshift-default-8687bbf6cc-p9mf7\" (UID: \"f464e173-8ada-4978-967f-7897d5bc5823\") " pod="openshift-ingress/e2e-api-a-25de17-openshift-default-8687bbf6cc-p9mf7"
Oct 30 22:13:12.071547 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:12.069331    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f464e173-8ada-4978-967f-7897d5bc5823-istio-podinfo\") pod \"e2e-api-a-25de17-openshift-default-8687bbf6cc-p9mf7\" (UID: \"f464e173-8ada-4978-967f-7897d5bc5823\") " pod="openshift-ingress/e2e-api-a-25de17-openshift-default-8687bbf6cc-p9mf7"
Oct 30 22:13:12.072391 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:12.072189    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cx2x4\" (UniqueName: \"kubernetes.io/projected/f464e173-8ada-4978-967f-7897d5bc5823-kube-api-access-cx2x4\") pod \"e2e-api-a-25de17-openshift-default-8687bbf6cc-p9mf7\" (UID: \"f464e173-8ada-4978-967f-7897d5bc5823\") " pod="openshift-ingress/e2e-api-a-25de17-openshift-default-8687bbf6cc-p9mf7"
Oct 30 22:13:12.149165 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:12.149104    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f464e173-8ada-4978-967f-7897d5bc5823-istio-token\") pod \"e2e-api-a-25de17-openshift-default-8687bbf6cc-p9mf7\" (UID: \"f464e173-8ada-4978-967f-7897d5bc5823\") " pod="openshift-ingress/e2e-api-a-25de17-openshift-default-8687bbf6cc-p9mf7"
Oct 30 22:13:12.403087 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:12.403049    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-25de17-openshift-default-8687bbf6cc-p9mf7"
Oct 30 22:13:12.827128 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:12.827096    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-25de17-openshift-default-8687bbf6cc-p9mf7"]
Oct 30 22:13:12.862353 ip-10-0-130-60 kubenswrapper[5061]: W1030 22:13:12.862321    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf464e173_8ada_4978_967f_7897d5bc5823.slice/crio-a5a72e0acfa770b686d4c6a7d9583167b409e494c7c403e073be9f7f25ce58ca WatchSource:0}: Error finding container a5a72e0acfa770b686d4c6a7d9583167b409e494c7c403e073be9f7f25ce58ca: Status 404 returned error can't find the container with id a5a72e0acfa770b686d4c6a7d9583167b409e494c7c403e073be9f7f25ce58ca
Oct 30 22:13:12.865217 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:12.865178    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:13:12.865333 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:12.865265    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:13:12.865333 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:12.865311    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:13:13.797207 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:13.797169    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-25de17-openshift-default-8687bbf6cc-p9mf7" event={"ID":"f464e173-8ada-4978-967f-7897d5bc5823","Type":"ContainerStarted","Data":"afb0f3ee46792d05f5b3ccc7c84b1d1ecdbe202f6943a3d166ff2be91f705a77"}
Oct 30 22:13:13.797392 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:13.797231    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-25de17-openshift-default-8687bbf6cc-p9mf7" event={"ID":"f464e173-8ada-4978-967f-7897d5bc5823","Type":"ContainerStarted","Data":"a5a72e0acfa770b686d4c6a7d9583167b409e494c7c403e073be9f7f25ce58ca"}
Oct 30 22:13:14.017600 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:14.017540    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-25de17-openshift-default-8687bbf6cc-p9mf7" podStartSLOduration=3.017508513 podStartE2EDuration="3.017508513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:13:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:13:13.969726832 +0000 UTC m=+1847.582496644" watchObservedRunningTime="2026-10-30 22:13:14.017508513 +0000 UTC m=+1847.630278272"
Oct 30 22:13:14.403515 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:14.403480    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-25de17-openshift-default-8687bbf6cc-p9mf7"
Oct 30 22:13:14.409175 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:14.409151    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-25de17-openshift-default-8687bbf6cc-p9mf7"
Oct 30 22:13:14.806082 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:14.806049    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-25de17-openshift-default-8687bbf6cc-p9mf7"
Oct 30 22:13:14.807067 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:14.807043    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-25de17-openshift-default-8687bbf6cc-p9mf7"
Oct 30 22:13:16.346794 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:16.346754    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-c9dc10-c94b4fcc5-5jrrn"]
Oct 30 22:13:16.351677 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:16.351649    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-c9dc10-c94b4fcc5-5jrrn"
Oct 30 22:13:16.401059 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:16.401028    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w6-c9dc10\""
Oct 30 22:13:16.423154 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:16.423105    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64ca325f-739e-4a19-b117-8ddff188ea10-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-c9dc10-c94b4fcc5-5jrrn\" (UID: \"64ca325f-739e-4a19-b117-8ddff188ea10\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-c9dc10-c94b4fcc5-5jrrn"
Oct 30 22:13:16.423353 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:16.423273    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hq5zd\" (UniqueName: \"kubernetes.io/projected/64ca325f-739e-4a19-b117-8ddff188ea10-kube-api-access-hq5zd\") pod \"payload-pre-processing-e2e-shared-b-w6-c9dc10-c94b4fcc5-5jrrn\" (UID: \"64ca325f-739e-4a19-b117-8ddff188ea10\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-c9dc10-c94b4fcc5-5jrrn"
Oct 30 22:13:16.452680 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:16.452644    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w6-c9dc10-dockercfg-rl4hw\""
Oct 30 22:13:16.494810 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:16.494777    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-c9dc10-c94b4fcc5-5jrrn"]
Oct 30 22:13:16.529739 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:16.529706    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64ca325f-739e-4a19-b117-8ddff188ea10-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-c9dc10-c94b4fcc5-5jrrn\" (UID: \"64ca325f-739e-4a19-b117-8ddff188ea10\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-c9dc10-c94b4fcc5-5jrrn"
Oct 30 22:13:16.530119 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:16.530098    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hq5zd\" (UniqueName: \"kubernetes.io/projected/64ca325f-739e-4a19-b117-8ddff188ea10-kube-api-access-hq5zd\") pod \"payload-pre-processing-e2e-shared-b-w6-c9dc10-c94b4fcc5-5jrrn\" (UID: \"64ca325f-739e-4a19-b117-8ddff188ea10\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-c9dc10-c94b4fcc5-5jrrn"
Oct 30 22:13:16.531392 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:16.531366    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64ca325f-739e-4a19-b117-8ddff188ea10-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-c9dc10-c94b4fcc5-5jrrn\" (UID: \"64ca325f-739e-4a19-b117-8ddff188ea10\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-c9dc10-c94b4fcc5-5jrrn"
Oct 30 22:13:16.609918 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:16.605056    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hq5zd\" (UniqueName: \"kubernetes.io/projected/64ca325f-739e-4a19-b117-8ddff188ea10-kube-api-access-hq5zd\") pod \"payload-pre-processing-e2e-shared-b-w6-c9dc10-c94b4fcc5-5jrrn\" (UID: \"64ca325f-739e-4a19-b117-8ddff188ea10\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-c9dc10-c94b4fcc5-5jrrn"
Oct 30 22:13:16.674569 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:16.673009    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-c9dc10-c94b4fcc5-5jrrn"
Oct 30 22:13:17.067946 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:17.064061    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-c9dc10-c94b4fcc5-5jrrn"]
Oct 30 22:13:17.832716 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:17.832683    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-c9dc10-c94b4fcc5-5jrrn" event={"ID":"64ca325f-739e-4a19-b117-8ddff188ea10","Type":"ContainerStarted","Data":"17260df1162d1e778a70113a6d4df9a6b12ba67df360c4b01832250d0c828e80"}
Oct 30 22:13:17.833109 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:17.832738    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-c9dc10-c94b4fcc5-5jrrn" event={"ID":"64ca325f-739e-4a19-b117-8ddff188ea10","Type":"ContainerStarted","Data":"54ffffff7dc888205757060c44a67a323fbfa6f8df706c4ccad53d6f1aec887c"}
Oct 30 22:13:17.833109 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:17.832805    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-c9dc10-c94b4fcc5-5jrrn"
Oct 30 22:13:18.629789 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:18.629733    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-c9dc10-c94b4fcc5-5jrrn" podStartSLOduration=2.62971812 podStartE2EDuration="2.62971812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:13:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:13:18.627514872 +0000 UTC m=+1852.240284608" watchObservedRunningTime="2026-10-30 22:13:18.62971812 +0000 UTC m=+1852.242487878"
Oct 30 22:13:23.412723 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:23.412684    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-25de17-757499fd87-8g2x5"]
Oct 30 22:13:23.417573 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:23.417516    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-25de17-757499fd87-8g2x5"
Oct 30 22:13:23.462374 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:23.462344    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5j8hs\" (UniqueName: \"kubernetes.io/projected/ef968170-02da-4a02-8182-8a5dbb14d62c-kube-api-access-5j8hs\") pod \"payload-pre-processing-e2e-api-a-25de17-757499fd87-8g2x5\" (UID: \"ef968170-02da-4a02-8182-8a5dbb14d62c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-25de17-757499fd87-8g2x5"
Oct 30 22:13:23.462596 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:23.462572    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef968170-02da-4a02-8182-8a5dbb14d62c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-25de17-757499fd87-8g2x5\" (UID: \"ef968170-02da-4a02-8182-8a5dbb14d62c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-25de17-757499fd87-8g2x5"
Oct 30 22:13:23.521548 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:23.517702    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-25de17-dockercfg-zmjm6\""
Oct 30 22:13:23.521548 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:23.518353    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-25de17\""
Oct 30 22:13:23.658629 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:23.656708    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5j8hs\" (UniqueName: \"kubernetes.io/projected/ef968170-02da-4a02-8182-8a5dbb14d62c-kube-api-access-5j8hs\") pod \"payload-pre-processing-e2e-api-a-25de17-757499fd87-8g2x5\" (UID: \"ef968170-02da-4a02-8182-8a5dbb14d62c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-25de17-757499fd87-8g2x5"
Oct 30 22:13:23.658629 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:23.657059    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef968170-02da-4a02-8182-8a5dbb14d62c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-25de17-757499fd87-8g2x5\" (UID: \"ef968170-02da-4a02-8182-8a5dbb14d62c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-25de17-757499fd87-8g2x5"
Oct 30 22:13:23.664602 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:23.662078    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef968170-02da-4a02-8182-8a5dbb14d62c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-25de17-757499fd87-8g2x5\" (UID: \"ef968170-02da-4a02-8182-8a5dbb14d62c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-25de17-757499fd87-8g2x5"
Oct 30 22:13:23.715602 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:23.691166    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-25de17-757499fd87-8g2x5"]
Oct 30 22:13:23.715602 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:23.691252    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-25de17-68d48cf4fd-hn6nb"]
Oct 30 22:13:23.721479 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:23.721435    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-25de17-68d48cf4fd-hn6nb"]
Oct 30 22:13:23.721904 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:23.721870    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-25de17-68d48cf4fd-hn6nb"
Oct 30 22:13:23.734573 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:23.733191    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5j8hs\" (UniqueName: \"kubernetes.io/projected/ef968170-02da-4a02-8182-8a5dbb14d62c-kube-api-access-5j8hs\") pod \"payload-pre-processing-e2e-api-a-25de17-757499fd87-8g2x5\" (UID: \"ef968170-02da-4a02-8182-8a5dbb14d62c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-25de17-757499fd87-8g2x5"
Oct 30 22:13:23.860307 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:23.860273    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slhqj\" (UniqueName: \"kubernetes.io/projected/b6c1f484-09fa-46b7-a171-d481e895efa9-kube-api-access-slhqj\") pod \"payload-processing-e2e-api-a-25de17-68d48cf4fd-hn6nb\" (UID: \"b6c1f484-09fa-46b7-a171-d481e895efa9\") " pod="openshift-ingress/payload-processing-e2e-api-a-25de17-68d48cf4fd-hn6nb"
Oct 30 22:13:23.860497 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:23.860330    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6c1f484-09fa-46b7-a171-d481e895efa9-plugins-config-volume\") pod \"payload-processing-e2e-api-a-25de17-68d48cf4fd-hn6nb\" (UID: \"b6c1f484-09fa-46b7-a171-d481e895efa9\") " pod="openshift-ingress/payload-processing-e2e-api-a-25de17-68d48cf4fd-hn6nb"
Oct 30 22:13:23.962291 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:23.962194    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slhqj\" (UniqueName: \"kubernetes.io/projected/b6c1f484-09fa-46b7-a171-d481e895efa9-kube-api-access-slhqj\") pod \"payload-processing-e2e-api-a-25de17-68d48cf4fd-hn6nb\" (UID: \"b6c1f484-09fa-46b7-a171-d481e895efa9\") " pod="openshift-ingress/payload-processing-e2e-api-a-25de17-68d48cf4fd-hn6nb"
Oct 30 22:13:23.962291 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:23.962274    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6c1f484-09fa-46b7-a171-d481e895efa9-plugins-config-volume\") pod \"payload-processing-e2e-api-a-25de17-68d48cf4fd-hn6nb\" (UID: \"b6c1f484-09fa-46b7-a171-d481e895efa9\") " pod="openshift-ingress/payload-processing-e2e-api-a-25de17-68d48cf4fd-hn6nb"
Oct 30 22:13:23.963461 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:23.963437    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6c1f484-09fa-46b7-a171-d481e895efa9-plugins-config-volume\") pod \"payload-processing-e2e-api-a-25de17-68d48cf4fd-hn6nb\" (UID: \"b6c1f484-09fa-46b7-a171-d481e895efa9\") " pod="openshift-ingress/payload-processing-e2e-api-a-25de17-68d48cf4fd-hn6nb"
Oct 30 22:13:24.017326 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:24.017298    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slhqj\" (UniqueName: \"kubernetes.io/projected/b6c1f484-09fa-46b7-a171-d481e895efa9-kube-api-access-slhqj\") pod \"payload-processing-e2e-api-a-25de17-68d48cf4fd-hn6nb\" (UID: \"b6c1f484-09fa-46b7-a171-d481e895efa9\") " pod="openshift-ingress/payload-processing-e2e-api-a-25de17-68d48cf4fd-hn6nb"
Oct 30 22:13:24.030970 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:24.030943    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-25de17-757499fd87-8g2x5"
Oct 30 22:13:24.055396 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:24.055373    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-25de17-68d48cf4fd-hn6nb"
Oct 30 22:13:24.414358 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:24.414315    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-25de17-757499fd87-8g2x5"]
Oct 30 22:13:24.450505 ip-10-0-130-60 kubenswrapper[5061]: W1030 22:13:24.450474    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef968170_02da_4a02_8182_8a5dbb14d62c.slice/crio-d8593939c92433cd2a02894644b808a1c66b487423b77d84ffa7f36e65ed6f31 WatchSource:0}: Error finding container d8593939c92433cd2a02894644b808a1c66b487423b77d84ffa7f36e65ed6f31: Status 404 returned error can't find the container with id d8593939c92433cd2a02894644b808a1c66b487423b77d84ffa7f36e65ed6f31
Oct 30 22:13:24.473577 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:24.473548    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-25de17-68d48cf4fd-hn6nb"]
Oct 30 22:13:24.668801 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:13:24.668712    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 22:13:24.883051 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:24.883006    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-25de17-68d48cf4fd-hn6nb" event={"ID":"b6c1f484-09fa-46b7-a171-d481e895efa9","Type":"ContainerStarted","Data":"749d5687071772c6312c4f7429423ff66e645428ae027df81e2d3a01f92e201b"}
Oct 30 22:13:24.883217 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:24.883080    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-25de17-68d48cf4fd-hn6nb"
Oct 30 22:13:24.883217 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:24.883100    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-25de17-68d48cf4fd-hn6nb" event={"ID":"b6c1f484-09fa-46b7-a171-d481e895efa9","Type":"ContainerStarted","Data":"32d535998ba22477462a966dd2723653a7a0747c38dd94cc1bf1253b00daa5be"}
Oct 30 22:13:24.884812 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:24.884783    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-25de17-757499fd87-8g2x5" event={"ID":"ef968170-02da-4a02-8182-8a5dbb14d62c","Type":"ContainerStarted","Data":"b5157211072ded63f5137411287ab05e92c7098bc4b2eebcec99ee7a47218e75"}
Oct 30 22:13:24.884936 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:24.884824    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-25de17-757499fd87-8g2x5" event={"ID":"ef968170-02da-4a02-8182-8a5dbb14d62c","Type":"ContainerStarted","Data":"d8593939c92433cd2a02894644b808a1c66b487423b77d84ffa7f36e65ed6f31"}
Oct 30 22:13:24.884936 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:24.884859    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-25de17-757499fd87-8g2x5"
Oct 30 22:13:25.199565 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:25.199491    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-25de17-68d48cf4fd-hn6nb" podStartSLOduration=2.199473362 podStartE2EDuration="2.199473362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:13:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:13:25.041741999 +0000 UTC m=+1858.654511776" watchObservedRunningTime="2026-10-30 22:13:25.199473362 +0000 UTC m=+1858.812243120"
Oct 30 22:13:25.201746 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:25.201705    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-25de17-757499fd87-8g2x5" podStartSLOduration=2.201691606 podStartE2EDuration="2.201691606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:13:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:13:25.151488769 +0000 UTC m=+1858.764258539" watchObservedRunningTime="2026-10-30 22:13:25.201691606 +0000 UTC m=+1858.814461364"
Oct 30 22:13:28.843412 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:28.843380    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-c9dc10-c94b4fcc5-5jrrn"
Oct 30 22:13:35.894500 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:35.894467    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-25de17-68d48cf4fd-hn6nb"
Oct 30 22:13:35.895123 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:35.894690    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-25de17-757499fd87-8g2x5"
Oct 30 22:13:44.800451 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:44.800418    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7d434591-bjs4d"]
Oct 30 22:13:44.846898 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:44.846863    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7d434591-bjs4d"]
Oct 30 22:13:46.530778 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:46.530733    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="202217c5-cd5c-4abd-8081-342d904e86f1" path="/var/lib/kubelet/pods/202217c5-cd5c-4abd-8081-342d904e86f1/volumes"
Oct 30 22:13:51.787010 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:51.786973    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-7d434591-openshift-default-58f74c4865-rm5js"]
Oct 30 22:13:51.787458 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:51.787195    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-7d434591-openshift-default-58f74c4865-rm5js" podUID="ce46d672-35a3-4cca-a9fc-b070939de95f" containerName="istio-proxy" containerID="cri-o://1ab601cd0d39dbde394f0282268e4927c3b47d76885539c00d042e65b53d13fc" gracePeriod=30
Oct 30 22:13:54.670266 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:13:54.670210    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 22:13:57.081184 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:57.081159    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7d434591-openshift-default-58f74c4865-rm5js"
Oct 30 22:13:57.224411 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:57.224376    5061 generic.go:356] "Generic (PLEG): container finished" podID="ce46d672-35a3-4cca-a9fc-b070939de95f" containerID="1ab601cd0d39dbde394f0282268e4927c3b47d76885539c00d042e65b53d13fc" exitCode=0
Oct 30 22:13:57.224602 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:57.224429    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7d434591-openshift-default-58f74c4865-rm5js"
Oct 30 22:13:57.224602 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:57.224463    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7d434591-openshift-default-58f74c4865-rm5js" event={"ID":"ce46d672-35a3-4cca-a9fc-b070939de95f","Type":"ContainerDied","Data":"1ab601cd0d39dbde394f0282268e4927c3b47d76885539c00d042e65b53d13fc"}
Oct 30 22:13:57.224602 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:57.224515    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7d434591-openshift-default-58f74c4865-rm5js" event={"ID":"ce46d672-35a3-4cca-a9fc-b070939de95f","Type":"ContainerDied","Data":"a750f773a54be887ded3179581f4bd3b96f4734168de0c23f099e67bcf841fe3"}
Oct 30 22:13:57.224602 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:57.224564    5061 scope.go:122] "RemoveContainer" containerID="1ab601cd0d39dbde394f0282268e4927c3b47d76885539c00d042e65b53d13fc"
Oct 30 22:13:57.235464 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:57.235447    5061 scope.go:122] "RemoveContainer" containerID="614f26a23cbceea26931c4ad079465625aa89f7b37390644c6d0999bb51b7970"
Oct 30 22:13:57.240835 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:57.240799    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ce46d672-35a3-4cca-a9fc-b070939de95f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce46d672-35a3-4cca-a9fc-b070939de95f-istio-podinfo\") pod \"ce46d672-35a3-4cca-a9fc-b070939de95f\" (UID: \"ce46d672-35a3-4cca-a9fc-b070939de95f\") "
Oct 30 22:13:57.241296 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:57.241274    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce46d672-35a3-4cca-a9fc-b070939de95f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce46d672-35a3-4cca-a9fc-b070939de95f-workload-socket\") pod \"ce46d672-35a3-4cca-a9fc-b070939de95f\" (UID: \"ce46d672-35a3-4cca-a9fc-b070939de95f\") "
Oct 30 22:13:57.241643 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:57.241314    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce46d672-35a3-4cca-a9fc-b070939de95f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce46d672-35a3-4cca-a9fc-b070939de95f-workload-certs\") pod \"ce46d672-35a3-4cca-a9fc-b070939de95f\" (UID: \"ce46d672-35a3-4cca-a9fc-b070939de95f\") "
Oct 30 22:13:57.241643 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:57.241423    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce46d672-35a3-4cca-a9fc-b070939de95f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce46d672-35a3-4cca-a9fc-b070939de95f-istio-envoy\") pod \"ce46d672-35a3-4cca-a9fc-b070939de95f\" (UID: \"ce46d672-35a3-4cca-a9fc-b070939de95f\") "
Oct 30 22:13:57.241643 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:57.241537    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce46d672-35a3-4cca-a9fc-b070939de95f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce46d672-35a3-4cca-a9fc-b070939de95f-istio-data\") pod \"ce46d672-35a3-4cca-a9fc-b070939de95f\" (UID: \"ce46d672-35a3-4cca-a9fc-b070939de95f\") "
Oct 30 22:13:57.241643 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:57.241586    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce46d672-35a3-4cca-a9fc-b070939de95f-kube-api-access-fgh9p\" (UniqueName: \"kubernetes.io/projected/ce46d672-35a3-4cca-a9fc-b070939de95f-kube-api-access-fgh9p\") pod \"ce46d672-35a3-4cca-a9fc-b070939de95f\" (UID: \"ce46d672-35a3-4cca-a9fc-b070939de95f\") "
Oct 30 22:13:57.241970 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:57.241659    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce46d672-35a3-4cca-a9fc-b070939de95f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce46d672-35a3-4cca-a9fc-b070939de95f-istiod-ca-cert\") pod \"ce46d672-35a3-4cca-a9fc-b070939de95f\" (UID: \"ce46d672-35a3-4cca-a9fc-b070939de95f\") "
Oct 30 22:13:57.241970 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:57.241758    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce46d672-35a3-4cca-a9fc-b070939de95f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce46d672-35a3-4cca-a9fc-b070939de95f-credential-socket\") pod \"ce46d672-35a3-4cca-a9fc-b070939de95f\" (UID: \"ce46d672-35a3-4cca-a9fc-b070939de95f\") "
Oct 30 22:13:57.241970 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:57.241763    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce46d672-35a3-4cca-a9fc-b070939de95f-workload-socket" pod "ce46d672-35a3-4cca-a9fc-b070939de95f" (UID: "ce46d672-35a3-4cca-a9fc-b070939de95f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:13:57.241970 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:57.241849    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce46d672-35a3-4cca-a9fc-b070939de95f-istio-token\" (UniqueName: \"kubernetes.io/projected/ce46d672-35a3-4cca-a9fc-b070939de95f-istio-token\") pod \"ce46d672-35a3-4cca-a9fc-b070939de95f\" (UID: \"ce46d672-35a3-4cca-a9fc-b070939de95f\") "
Oct 30 22:13:57.242365 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:57.242330    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce46d672-35a3-4cca-a9fc-b070939de95f-credential-socket" pod "ce46d672-35a3-4cca-a9fc-b070939de95f" (UID: "ce46d672-35a3-4cca-a9fc-b070939de95f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:13:57.242603 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:57.242579    5061 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce46d672-35a3-4cca-a9fc-b070939de95f-credential-socket\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:13:57.242679 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:57.242607    5061 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce46d672-35a3-4cca-a9fc-b070939de95f-workload-socket\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:13:57.242806 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:57.242783    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce46d672-35a3-4cca-a9fc-b070939de95f-istiod-ca-cert" pod "ce46d672-35a3-4cca-a9fc-b070939de95f" (UID: "ce46d672-35a3-4cca-a9fc-b070939de95f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:13:57.242873 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:57.242853    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce46d672-35a3-4cca-a9fc-b070939de95f-workload-certs" pod "ce46d672-35a3-4cca-a9fc-b070939de95f" (UID: "ce46d672-35a3-4cca-a9fc-b070939de95f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:13:57.243232 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:57.243208    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce46d672-35a3-4cca-a9fc-b070939de95f-istio-data" pod "ce46d672-35a3-4cca-a9fc-b070939de95f" (UID: "ce46d672-35a3-4cca-a9fc-b070939de95f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:13:57.249266 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:57.249064    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce46d672-35a3-4cca-a9fc-b070939de95f-istio-envoy" pod "ce46d672-35a3-4cca-a9fc-b070939de95f" (UID: "ce46d672-35a3-4cca-a9fc-b070939de95f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:13:57.249266 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:57.249166    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce46d672-35a3-4cca-a9fc-b070939de95f-istio-token" pod "ce46d672-35a3-4cca-a9fc-b070939de95f" (UID: "ce46d672-35a3-4cca-a9fc-b070939de95f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:13:57.249266 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:57.249230    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce46d672-35a3-4cca-a9fc-b070939de95f-kube-api-access-fgh9p" pod "ce46d672-35a3-4cca-a9fc-b070939de95f" (UID: "ce46d672-35a3-4cca-a9fc-b070939de95f"). InnerVolumeSpecName "kube-api-access-fgh9p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:13:57.249652 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:57.249635    5061 scope.go:122] "RemoveContainer" containerID="1ab601cd0d39dbde394f0282268e4927c3b47d76885539c00d042e65b53d13fc"
Oct 30 22:13:57.250144 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:13:57.250115    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ab601cd0d39dbde394f0282268e4927c3b47d76885539c00d042e65b53d13fc\": container with ID starting with 1ab601cd0d39dbde394f0282268e4927c3b47d76885539c00d042e65b53d13fc not found: ID does not exist" containerID="1ab601cd0d39dbde394f0282268e4927c3b47d76885539c00d042e65b53d13fc"
Oct 30 22:13:57.250255 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:57.250151    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ab601cd0d39dbde394f0282268e4927c3b47d76885539c00d042e65b53d13fc"} err="failed to get container status \"1ab601cd0d39dbde394f0282268e4927c3b47d76885539c00d042e65b53d13fc\": rpc error: code = NotFound desc = could not find container \"1ab601cd0d39dbde394f0282268e4927c3b47d76885539c00d042e65b53d13fc\": container with ID starting with 1ab601cd0d39dbde394f0282268e4927c3b47d76885539c00d042e65b53d13fc not found: ID does not exist"
Oct 30 22:13:57.250255 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:57.250175    5061 scope.go:122] "RemoveContainer" containerID="614f26a23cbceea26931c4ad079465625aa89f7b37390644c6d0999bb51b7970"
Oct 30 22:13:57.250582 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:13:57.250515    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"614f26a23cbceea26931c4ad079465625aa89f7b37390644c6d0999bb51b7970\": container with ID starting with 614f26a23cbceea26931c4ad079465625aa89f7b37390644c6d0999bb51b7970 not found: ID does not exist" containerID="614f26a23cbceea26931c4ad079465625aa89f7b37390644c6d0999bb51b7970"
Oct 30 22:13:57.250673 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:57.250593    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"614f26a23cbceea26931c4ad079465625aa89f7b37390644c6d0999bb51b7970"} err="failed to get container status \"614f26a23cbceea26931c4ad079465625aa89f7b37390644c6d0999bb51b7970\": rpc error: code = NotFound desc = could not find container \"614f26a23cbceea26931c4ad079465625aa89f7b37390644c6d0999bb51b7970\": container with ID starting with 614f26a23cbceea26931c4ad079465625aa89f7b37390644c6d0999bb51b7970 not found: ID does not exist"
Oct 30 22:13:57.252691 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:57.252667    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ce46d672-35a3-4cca-a9fc-b070939de95f-istio-podinfo" pod "ce46d672-35a3-4cca-a9fc-b070939de95f" (UID: "ce46d672-35a3-4cca-a9fc-b070939de95f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:13:57.344339 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:57.344272    5061 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce46d672-35a3-4cca-a9fc-b070939de95f-istio-data\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:13:57.344339 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:57.344298    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgh9p\" (UniqueName: \"kubernetes.io/projected/ce46d672-35a3-4cca-a9fc-b070939de95f-kube-api-access-fgh9p\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:13:57.344339 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:57.344312    5061 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce46d672-35a3-4cca-a9fc-b070939de95f-istiod-ca-cert\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:13:57.344339 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:57.344325    5061 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce46d672-35a3-4cca-a9fc-b070939de95f-istio-token\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:13:57.344339 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:57.344336    5061 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce46d672-35a3-4cca-a9fc-b070939de95f-istio-podinfo\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:13:57.344608 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:57.344345    5061 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce46d672-35a3-4cca-a9fc-b070939de95f-workload-certs\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:13:57.344608 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:57.344353    5061 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce46d672-35a3-4cca-a9fc-b070939de95f-istio-envoy\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:13:57.642394 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:57.642268    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-7d434591-openshift-default-58f74c4865-rm5js"]
Oct 30 22:13:57.693376 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:57.693347    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-7d434591-openshift-default-58f74c4865-rm5js"]
Oct 30 22:13:58.271376 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:58.271333    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-411ff1-794dbcbd69-sfhsl"]
Oct 30 22:13:58.272287 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:58.272264    5061 cpu_manager.go:414] "containerMap: removing container" podUID="ce46d672-35a3-4cca-a9fc-b070939de95f" containerName="istio-proxy"
Oct 30 22:13:58.272287 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:58.272286    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce46d672-35a3-4cca-a9fc-b070939de95f" containerName="istio-proxy"
Oct 30 22:13:58.272472 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:58.272305    5061 cpu_manager.go:414] "containerMap: removing container" podUID="ce46d672-35a3-4cca-a9fc-b070939de95f" containerName="istio-proxy"
Oct 30 22:13:58.272472 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:58.272311    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce46d672-35a3-4cca-a9fc-b070939de95f" containerName="istio-proxy"
Oct 30 22:13:58.272596 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:58.272490    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce46d672-35a3-4cca-a9fc-b070939de95f" containerName="istio-proxy"
Oct 30 22:13:58.272596 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:58.272509    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce46d672-35a3-4cca-a9fc-b070939de95f" containerName="istio-proxy"
Oct 30 22:13:58.277508 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:58.277484    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-411ff1-794dbcbd69-sfhsl"
Oct 30 22:13:58.327007 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:58.326982    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w1-411ff1-serving-cert\""
Oct 30 22:13:58.380280 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:58.380246    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-411ff1-794dbcbd69-sfhsl"]
Oct 30 22:13:58.459403 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:58.459369    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6t4fs\" (UniqueName: \"kubernetes.io/projected/2c04bb48-1286-48fe-af0d-fc9ec478e663-kube-api-access-6t4fs\") pod \"maas-api-e2e-shared-b-w1-411ff1-794dbcbd69-sfhsl\" (UID: \"2c04bb48-1286-48fe-af0d-fc9ec478e663\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-411ff1-794dbcbd69-sfhsl"
Oct 30 22:13:58.459586 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:58.459539    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c04bb48-1286-48fe-af0d-fc9ec478e663-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-411ff1-794dbcbd69-sfhsl\" (UID: \"2c04bb48-1286-48fe-af0d-fc9ec478e663\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-411ff1-794dbcbd69-sfhsl"
Oct 30 22:13:58.459637 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:58.459590    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c04bb48-1286-48fe-af0d-fc9ec478e663-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-411ff1-794dbcbd69-sfhsl\" (UID: \"2c04bb48-1286-48fe-af0d-fc9ec478e663\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-411ff1-794dbcbd69-sfhsl"
Oct 30 22:13:58.459746 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:58.459733    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c04bb48-1286-48fe-af0d-fc9ec478e663-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-411ff1-794dbcbd69-sfhsl\" (UID: \"2c04bb48-1286-48fe-af0d-fc9ec478e663\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-411ff1-794dbcbd69-sfhsl"
Oct 30 22:13:58.530055 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:58.529979    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce46d672-35a3-4cca-a9fc-b070939de95f" path="/var/lib/kubelet/pods/ce46d672-35a3-4cca-a9fc-b070939de95f/volumes"
Oct 30 22:13:58.560968 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:58.560936    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c04bb48-1286-48fe-af0d-fc9ec478e663-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-411ff1-794dbcbd69-sfhsl\" (UID: \"2c04bb48-1286-48fe-af0d-fc9ec478e663\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-411ff1-794dbcbd69-sfhsl"
Oct 30 22:13:58.561125 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:58.561041    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c04bb48-1286-48fe-af0d-fc9ec478e663-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-411ff1-794dbcbd69-sfhsl\" (UID: \"2c04bb48-1286-48fe-af0d-fc9ec478e663\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-411ff1-794dbcbd69-sfhsl"
Oct 30 22:13:58.561125 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:58.561098    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6t4fs\" (UniqueName: \"kubernetes.io/projected/2c04bb48-1286-48fe-af0d-fc9ec478e663-kube-api-access-6t4fs\") pod \"maas-api-e2e-shared-b-w1-411ff1-794dbcbd69-sfhsl\" (UID: \"2c04bb48-1286-48fe-af0d-fc9ec478e663\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-411ff1-794dbcbd69-sfhsl"
Oct 30 22:13:58.561224 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:58.561173    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c04bb48-1286-48fe-af0d-fc9ec478e663-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-411ff1-794dbcbd69-sfhsl\" (UID: \"2c04bb48-1286-48fe-af0d-fc9ec478e663\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-411ff1-794dbcbd69-sfhsl"
Oct 30 22:13:58.561850 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:58.561828    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c04bb48-1286-48fe-af0d-fc9ec478e663-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-411ff1-794dbcbd69-sfhsl\" (UID: \"2c04bb48-1286-48fe-af0d-fc9ec478e663\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-411ff1-794dbcbd69-sfhsl"
Oct 30 22:13:58.563582 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:58.563520    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c04bb48-1286-48fe-af0d-fc9ec478e663-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-411ff1-794dbcbd69-sfhsl\" (UID: \"2c04bb48-1286-48fe-af0d-fc9ec478e663\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-411ff1-794dbcbd69-sfhsl"
Oct 30 22:13:58.563682 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:58.563617    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c04bb48-1286-48fe-af0d-fc9ec478e663-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-411ff1-794dbcbd69-sfhsl\" (UID: \"2c04bb48-1286-48fe-af0d-fc9ec478e663\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-411ff1-794dbcbd69-sfhsl"
Oct 30 22:13:58.615141 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:58.615115    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6t4fs\" (UniqueName: \"kubernetes.io/projected/2c04bb48-1286-48fe-af0d-fc9ec478e663-kube-api-access-6t4fs\") pod \"maas-api-e2e-shared-b-w1-411ff1-794dbcbd69-sfhsl\" (UID: \"2c04bb48-1286-48fe-af0d-fc9ec478e663\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-411ff1-794dbcbd69-sfhsl"
Oct 30 22:13:58.891894 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:58.891865    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-411ff1-794dbcbd69-sfhsl"
Oct 30 22:13:59.266494 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:59.266464    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-411ff1-794dbcbd69-sfhsl"]
Oct 30 22:13:59.267121 ip-10-0-130-60 kubenswrapper[5061]: W1030 22:13:59.267078    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c04bb48_1286_48fe_af0d_fc9ec478e663.slice/crio-e5172fdc4012a4e44eff71d880a16081c647ae8c795a431f004b8ecf7f5b8e8b WatchSource:0}: Error finding container e5172fdc4012a4e44eff71d880a16081c647ae8c795a431f004b8ecf7f5b8e8b: Status 404 returned error can't find the container with id e5172fdc4012a4e44eff71d880a16081c647ae8c795a431f004b8ecf7f5b8e8b
Oct 30 22:13:59.268768 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:13:59.268749    5061 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 22:14:00.248673 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:00.248642    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-411ff1-794dbcbd69-sfhsl" event={"ID":"2c04bb48-1286-48fe-af0d-fc9ec478e663","Type":"ContainerStarted","Data":"4ec166df46a0650130217fff82aeceee87af729bcce56d41cc3dad535784fbd4"}
Oct 30 22:14:00.249019 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:00.248701    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-411ff1-794dbcbd69-sfhsl" event={"ID":"2c04bb48-1286-48fe-af0d-fc9ec478e663","Type":"ContainerStarted","Data":"e5172fdc4012a4e44eff71d880a16081c647ae8c795a431f004b8ecf7f5b8e8b"}
Oct 30 22:14:00.249019 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:00.248740    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-411ff1-794dbcbd69-sfhsl"
Oct 30 22:14:00.381320 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:00.381266    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-411ff1-794dbcbd69-sfhsl" podStartSLOduration=1.5015055400000001 podStartE2EDuration="2.381253482s" totalImagesPullingTime="879.747942ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:13:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:14:00.37812598 +0000 UTC m=+1893.990895758" watchObservedRunningTime="2026-10-30 22:14:00.381253482 +0000 UTC m=+1893.994023240"
Oct 30 22:14:02.453574 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:02.450351    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-263513-ccd8b86f5-5sphw"]
Oct 30 22:14:02.457550 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:02.456123    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-263513-ccd8b86f5-5sphw"
Oct 30 22:14:02.566250 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:02.566209    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-263513\""
Oct 30 22:14:02.566456 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:02.566339    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-263513-dockercfg-ks54g\""
Oct 30 22:14:02.606013 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:02.605971    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-263513-ccd8b86f5-5sphw"]
Oct 30 22:14:02.659593 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:02.626102    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-263513-7c6995fd7b-4tgq8"]
Oct 30 22:14:02.659593 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:02.633761    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-263513-7c6995fd7b-4tgq8"
Oct 30 22:14:02.659593 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:02.635981    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffa04d4a-c607-46a7-b4e0-c675ff341103-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-263513-ccd8b86f5-5sphw\" (UID: \"ffa04d4a-c607-46a7-b4e0-c675ff341103\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-263513-ccd8b86f5-5sphw"
Oct 30 22:14:02.659593 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:02.636264    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2b7t8\" (UniqueName: \"kubernetes.io/projected/ffa04d4a-c607-46a7-b4e0-c675ff341103-kube-api-access-2b7t8\") pod \"payload-pre-processing-e2e-inf-a-263513-ccd8b86f5-5sphw\" (UID: \"ffa04d4a-c607-46a7-b4e0-c675ff341103\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-263513-ccd8b86f5-5sphw"
Oct 30 22:14:02.669551 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:02.666929    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-263513-7c6995fd7b-4tgq8"]
Oct 30 22:14:02.738610 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:02.737745    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2b7t8\" (UniqueName: \"kubernetes.io/projected/ffa04d4a-c607-46a7-b4e0-c675ff341103-kube-api-access-2b7t8\") pod \"payload-pre-processing-e2e-inf-a-263513-ccd8b86f5-5sphw\" (UID: \"ffa04d4a-c607-46a7-b4e0-c675ff341103\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-263513-ccd8b86f5-5sphw"
Oct 30 22:14:02.738610 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:02.738042    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffa04d4a-c607-46a7-b4e0-c675ff341103-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-263513-ccd8b86f5-5sphw\" (UID: \"ffa04d4a-c607-46a7-b4e0-c675ff341103\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-263513-ccd8b86f5-5sphw"
Oct 30 22:14:02.738610 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:02.738123    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcbrq\" (UniqueName: \"kubernetes.io/projected/3fe1bb49-4bcc-49bd-9caa-70f83cbe8715-kube-api-access-hcbrq\") pod \"payload-processing-e2e-inf-a-263513-7c6995fd7b-4tgq8\" (UID: \"3fe1bb49-4bcc-49bd-9caa-70f83cbe8715\") " pod="openshift-ingress/payload-processing-e2e-inf-a-263513-7c6995fd7b-4tgq8"
Oct 30 22:14:02.738610 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:02.738190    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fe1bb49-4bcc-49bd-9caa-70f83cbe8715-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-263513-7c6995fd7b-4tgq8\" (UID: \"3fe1bb49-4bcc-49bd-9caa-70f83cbe8715\") " pod="openshift-ingress/payload-processing-e2e-inf-a-263513-7c6995fd7b-4tgq8"
Oct 30 22:14:02.745018 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:02.739783    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffa04d4a-c607-46a7-b4e0-c675ff341103-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-263513-ccd8b86f5-5sphw\" (UID: \"ffa04d4a-c607-46a7-b4e0-c675ff341103\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-263513-ccd8b86f5-5sphw"
Oct 30 22:14:02.792654 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:02.792621    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2b7t8\" (UniqueName: \"kubernetes.io/projected/ffa04d4a-c607-46a7-b4e0-c675ff341103-kube-api-access-2b7t8\") pod \"payload-pre-processing-e2e-inf-a-263513-ccd8b86f5-5sphw\" (UID: \"ffa04d4a-c607-46a7-b4e0-c675ff341103\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-263513-ccd8b86f5-5sphw"
Oct 30 22:14:02.839622 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:02.839589    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcbrq\" (UniqueName: \"kubernetes.io/projected/3fe1bb49-4bcc-49bd-9caa-70f83cbe8715-kube-api-access-hcbrq\") pod \"payload-processing-e2e-inf-a-263513-7c6995fd7b-4tgq8\" (UID: \"3fe1bb49-4bcc-49bd-9caa-70f83cbe8715\") " pod="openshift-ingress/payload-processing-e2e-inf-a-263513-7c6995fd7b-4tgq8"
Oct 30 22:14:02.839791 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:02.839642    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fe1bb49-4bcc-49bd-9caa-70f83cbe8715-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-263513-7c6995fd7b-4tgq8\" (UID: \"3fe1bb49-4bcc-49bd-9caa-70f83cbe8715\") " pod="openshift-ingress/payload-processing-e2e-inf-a-263513-7c6995fd7b-4tgq8"
Oct 30 22:14:02.840404 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:02.840384    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fe1bb49-4bcc-49bd-9caa-70f83cbe8715-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-263513-7c6995fd7b-4tgq8\" (UID: \"3fe1bb49-4bcc-49bd-9caa-70f83cbe8715\") " pod="openshift-ingress/payload-processing-e2e-inf-a-263513-7c6995fd7b-4tgq8"
Oct 30 22:14:02.894133 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:02.894100    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcbrq\" (UniqueName: \"kubernetes.io/projected/3fe1bb49-4bcc-49bd-9caa-70f83cbe8715-kube-api-access-hcbrq\") pod \"payload-processing-e2e-inf-a-263513-7c6995fd7b-4tgq8\" (UID: \"3fe1bb49-4bcc-49bd-9caa-70f83cbe8715\") " pod="openshift-ingress/payload-processing-e2e-inf-a-263513-7c6995fd7b-4tgq8"
Oct 30 22:14:03.001520 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:03.001436    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-263513-7c6995fd7b-4tgq8"
Oct 30 22:14:03.083254 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:03.081945    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-263513-ccd8b86f5-5sphw"
Oct 30 22:14:03.374042 ip-10-0-130-60 kubenswrapper[5061]: W1030 22:14:03.374013    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fe1bb49_4bcc_49bd_9caa_70f83cbe8715.slice/crio-40b6a4dd7aba5c4f03bde365ae88bbd917a26bbe29323df5fbd6bcccefdaed8c WatchSource:0}: Error finding container 40b6a4dd7aba5c4f03bde365ae88bbd917a26bbe29323df5fbd6bcccefdaed8c: Status 404 returned error can't find the container with id 40b6a4dd7aba5c4f03bde365ae88bbd917a26bbe29323df5fbd6bcccefdaed8c
Oct 30 22:14:03.374431 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:03.374410    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-263513-7c6995fd7b-4tgq8"]
Oct 30 22:14:03.483177 ip-10-0-130-60 kubenswrapper[5061]: W1030 22:14:03.482272    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffa04d4a_c607_46a7_b4e0_c675ff341103.slice/crio-e6d4871aa99ceefccc4b28f42dd7c91f51d64659fabaf335779116fb57ffc428 WatchSource:0}: Error finding container e6d4871aa99ceefccc4b28f42dd7c91f51d64659fabaf335779116fb57ffc428: Status 404 returned error can't find the container with id e6d4871aa99ceefccc4b28f42dd7c91f51d64659fabaf335779116fb57ffc428
Oct 30 22:14:03.491564 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:03.491518    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-263513-ccd8b86f5-5sphw"]
Oct 30 22:14:04.297812 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:04.297760    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-263513-ccd8b86f5-5sphw" event={"ID":"ffa04d4a-c607-46a7-b4e0-c675ff341103","Type":"ContainerStarted","Data":"dcfd70860de19211172b564af88a4b0c6b19416ac141bd0b5d035f6b7c94a038"}
Oct 30 22:14:04.298001 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:04.297878    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-263513-ccd8b86f5-5sphw" event={"ID":"ffa04d4a-c607-46a7-b4e0-c675ff341103","Type":"ContainerStarted","Data":"e6d4871aa99ceefccc4b28f42dd7c91f51d64659fabaf335779116fb57ffc428"}
Oct 30 22:14:04.298001 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:04.297910    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-263513-ccd8b86f5-5sphw"
Oct 30 22:14:04.299401 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:04.299372    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-263513-7c6995fd7b-4tgq8" event={"ID":"3fe1bb49-4bcc-49bd-9caa-70f83cbe8715","Type":"ContainerStarted","Data":"444c6b807a34a63ca6c551ff9e73e90330ac21c9504e1320a400ce401cb81f46"}
Oct 30 22:14:04.299547 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:04.299423    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-263513-7c6995fd7b-4tgq8" event={"ID":"3fe1bb49-4bcc-49bd-9caa-70f83cbe8715","Type":"ContainerStarted","Data":"40b6a4dd7aba5c4f03bde365ae88bbd917a26bbe29323df5fbd6bcccefdaed8c"}
Oct 30 22:14:04.299547 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:04.299454    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-263513-7c6995fd7b-4tgq8"
Oct 30 22:14:04.409084 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:04.409026    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-263513-ccd8b86f5-5sphw" podStartSLOduration=2.409012278 podStartE2EDuration="2.409012278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:14:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:14:04.406792591 +0000 UTC m=+1898.019562350" watchObservedRunningTime="2026-10-30 22:14:04.409012278 +0000 UTC m=+1898.021782035"
Oct 30 22:14:04.558057 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:04.557936    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-263513-7c6995fd7b-4tgq8" podStartSLOduration=2.557915889 podStartE2EDuration="2.557915889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:14:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:14:04.51586977 +0000 UTC m=+1898.128639528" watchObservedRunningTime="2026-10-30 22:14:04.557915889 +0000 UTC m=+1898.170685648"
Oct 30 22:14:06.274131 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:06.274103    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-411ff1-794dbcbd69-sfhsl"
Oct 30 22:14:13.379650 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:13.379560    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-bvtk8"]
Oct 30 22:14:13.380222 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:13.379838    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-bvtk8" podUID="384f8a66-b2a4-4481-9416-00443b153db9" containerName="istio-proxy" containerID="cri-o://f4b6e4efb8ea79279731a9766205fd5a2d69a440b4127d3cc35e63300363e1d4" gracePeriod=30
Oct 30 22:14:15.310003 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:15.309975    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-263513-ccd8b86f5-5sphw"
Oct 30 22:14:15.310447 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:15.310083    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-263513-7c6995fd7b-4tgq8"
Oct 30 22:14:16.047300 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:16.047265    5061 patch_prober.go:29] interesting pod/e2e-worker-w3-f776cb-openshift-default-74d577586f-bvtk8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:14:16.047480 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:16.047328    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-bvtk8" podUID="384f8a66-b2a4-4481-9416-00443b153db9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:14:18.700779 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:18.700748    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-bvtk8"
Oct 30 22:14:18.880002 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:18.879965    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/384f8a66-b2a4-4481-9416-00443b153db9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/384f8a66-b2a4-4481-9416-00443b153db9-workload-socket\") pod \"384f8a66-b2a4-4481-9416-00443b153db9\" (UID: \"384f8a66-b2a4-4481-9416-00443b153db9\") "
Oct 30 22:14:18.880195 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:18.880026    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/384f8a66-b2a4-4481-9416-00443b153db9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/384f8a66-b2a4-4481-9416-00443b153db9-istio-podinfo\") pod \"384f8a66-b2a4-4481-9416-00443b153db9\" (UID: \"384f8a66-b2a4-4481-9416-00443b153db9\") "
Oct 30 22:14:18.880195 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:18.880056    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/384f8a66-b2a4-4481-9416-00443b153db9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/384f8a66-b2a4-4481-9416-00443b153db9-istio-envoy\") pod \"384f8a66-b2a4-4481-9416-00443b153db9\" (UID: \"384f8a66-b2a4-4481-9416-00443b153db9\") "
Oct 30 22:14:18.880336 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:18.880255    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/384f8a66-b2a4-4481-9416-00443b153db9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/384f8a66-b2a4-4481-9416-00443b153db9-credential-socket\") pod \"384f8a66-b2a4-4481-9416-00443b153db9\" (UID: \"384f8a66-b2a4-4481-9416-00443b153db9\") "
Oct 30 22:14:18.880336 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:18.880291    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/384f8a66-b2a4-4481-9416-00443b153db9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/384f8a66-b2a4-4481-9416-00443b153db9-istio-data\") pod \"384f8a66-b2a4-4481-9416-00443b153db9\" (UID: \"384f8a66-b2a4-4481-9416-00443b153db9\") "
Oct 30 22:14:18.880336 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:18.880314    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/384f8a66-b2a4-4481-9416-00443b153db9-workload-socket" pod "384f8a66-b2a4-4481-9416-00443b153db9" (UID: "384f8a66-b2a4-4481-9416-00443b153db9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:18.880471 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:18.880366    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/384f8a66-b2a4-4481-9416-00443b153db9-kube-api-access-hhdkb\" (UniqueName: \"kubernetes.io/projected/384f8a66-b2a4-4481-9416-00443b153db9-kube-api-access-hhdkb\") pod \"384f8a66-b2a4-4481-9416-00443b153db9\" (UID: \"384f8a66-b2a4-4481-9416-00443b153db9\") "
Oct 30 22:14:18.880471 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:18.880439    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/384f8a66-b2a4-4481-9416-00443b153db9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/384f8a66-b2a4-4481-9416-00443b153db9-istiod-ca-cert\") pod \"384f8a66-b2a4-4481-9416-00443b153db9\" (UID: \"384f8a66-b2a4-4481-9416-00443b153db9\") "
Oct 30 22:14:18.880606 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:18.880489    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/384f8a66-b2a4-4481-9416-00443b153db9-istio-token\" (UniqueName: \"kubernetes.io/projected/384f8a66-b2a4-4481-9416-00443b153db9-istio-token\") pod \"384f8a66-b2a4-4481-9416-00443b153db9\" (UID: \"384f8a66-b2a4-4481-9416-00443b153db9\") "
Oct 30 22:14:18.880661 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:18.880604    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/384f8a66-b2a4-4481-9416-00443b153db9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/384f8a66-b2a4-4481-9416-00443b153db9-workload-certs\") pod \"384f8a66-b2a4-4481-9416-00443b153db9\" (UID: \"384f8a66-b2a4-4481-9416-00443b153db9\") "
Oct 30 22:14:18.880661 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:18.880617    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/384f8a66-b2a4-4481-9416-00443b153db9-istio-data" pod "384f8a66-b2a4-4481-9416-00443b153db9" (UID: "384f8a66-b2a4-4481-9416-00443b153db9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:18.880832 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:18.880806    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/384f8a66-b2a4-4481-9416-00443b153db9-workload-certs" pod "384f8a66-b2a4-4481-9416-00443b153db9" (UID: "384f8a66-b2a4-4481-9416-00443b153db9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:18.880926 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:18.880854    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/384f8a66-b2a4-4481-9416-00443b153db9-istiod-ca-cert" pod "384f8a66-b2a4-4481-9416-00443b153db9" (UID: "384f8a66-b2a4-4481-9416-00443b153db9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:14:18.880926 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:18.880566    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/384f8a66-b2a4-4481-9416-00443b153db9-credential-socket" pod "384f8a66-b2a4-4481-9416-00443b153db9" (UID: "384f8a66-b2a4-4481-9416-00443b153db9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:18.881466 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:18.881438    5061 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/384f8a66-b2a4-4481-9416-00443b153db9-workload-socket\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:14:18.881634 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:18.881470    5061 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/384f8a66-b2a4-4481-9416-00443b153db9-credential-socket\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:14:18.881634 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:18.881485    5061 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/384f8a66-b2a4-4481-9416-00443b153db9-istio-data\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:14:18.881634 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:18.881498    5061 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/384f8a66-b2a4-4481-9416-00443b153db9-istiod-ca-cert\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:14:18.881634 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:18.881512    5061 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/384f8a66-b2a4-4481-9416-00443b153db9-workload-certs\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:14:18.882707 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:18.882684    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/384f8a66-b2a4-4481-9416-00443b153db9-istio-envoy" pod "384f8a66-b2a4-4481-9416-00443b153db9" (UID: "384f8a66-b2a4-4481-9416-00443b153db9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:18.882802 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:18.882689    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/384f8a66-b2a4-4481-9416-00443b153db9-istio-token" pod "384f8a66-b2a4-4481-9416-00443b153db9" (UID: "384f8a66-b2a4-4481-9416-00443b153db9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:14:18.882802 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:18.882744    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/384f8a66-b2a4-4481-9416-00443b153db9-istio-podinfo" pod "384f8a66-b2a4-4481-9416-00443b153db9" (UID: "384f8a66-b2a4-4481-9416-00443b153db9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:14:18.882876 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:18.882816    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/384f8a66-b2a4-4481-9416-00443b153db9-kube-api-access-hhdkb" pod "384f8a66-b2a4-4481-9416-00443b153db9" (UID: "384f8a66-b2a4-4481-9416-00443b153db9"). InnerVolumeSpecName "kube-api-access-hhdkb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:14:18.982945 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:18.982851    5061 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/384f8a66-b2a4-4481-9416-00443b153db9-istio-podinfo\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:14:18.982945 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:18.982882    5061 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/384f8a66-b2a4-4481-9416-00443b153db9-istio-envoy\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:14:18.982945 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:18.982895    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhdkb\" (UniqueName: \"kubernetes.io/projected/384f8a66-b2a4-4481-9416-00443b153db9-kube-api-access-hhdkb\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:14:18.982945 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:18.982906    5061 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/384f8a66-b2a4-4481-9416-00443b153db9-istio-token\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:14:19.449475 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:19.449438    5061 generic.go:356] "Generic (PLEG): container finished" podID="384f8a66-b2a4-4481-9416-00443b153db9" containerID="f4b6e4efb8ea79279731a9766205fd5a2d69a440b4127d3cc35e63300363e1d4" exitCode=0
Oct 30 22:14:19.449676 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:19.449488    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-bvtk8"
Oct 30 22:14:19.449676 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:19.449557    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-bvtk8" event={"ID":"384f8a66-b2a4-4481-9416-00443b153db9","Type":"ContainerDied","Data":"f4b6e4efb8ea79279731a9766205fd5a2d69a440b4127d3cc35e63300363e1d4"}
Oct 30 22:14:19.449676 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:19.449622    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-bvtk8" event={"ID":"384f8a66-b2a4-4481-9416-00443b153db9","Type":"ContainerDied","Data":"9ba2aa383fdf78a94cb0455b63ec24710cc02e96eaed986f4b5456db5ba20e4f"}
Oct 30 22:14:19.449676 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:19.449646    5061 scope.go:122] "RemoveContainer" containerID="f4b6e4efb8ea79279731a9766205fd5a2d69a440b4127d3cc35e63300363e1d4"
Oct 30 22:14:19.463189 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:19.463171    5061 scope.go:122] "RemoveContainer" containerID="f4b6e4efb8ea79279731a9766205fd5a2d69a440b4127d3cc35e63300363e1d4"
Oct 30 22:14:19.463460 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:14:19.463436    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4b6e4efb8ea79279731a9766205fd5a2d69a440b4127d3cc35e63300363e1d4\": container with ID starting with f4b6e4efb8ea79279731a9766205fd5a2d69a440b4127d3cc35e63300363e1d4 not found: ID does not exist" containerID="f4b6e4efb8ea79279731a9766205fd5a2d69a440b4127d3cc35e63300363e1d4"
Oct 30 22:14:19.463517 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:19.463469    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4b6e4efb8ea79279731a9766205fd5a2d69a440b4127d3cc35e63300363e1d4"} err="failed to get container status \"f4b6e4efb8ea79279731a9766205fd5a2d69a440b4127d3cc35e63300363e1d4\": rpc error: code = NotFound desc = could not find container \"f4b6e4efb8ea79279731a9766205fd5a2d69a440b4127d3cc35e63300363e1d4\": container with ID starting with f4b6e4efb8ea79279731a9766205fd5a2d69a440b4127d3cc35e63300363e1d4 not found: ID does not exist"
Oct 30 22:14:19.655315 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:19.655235    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-bvtk8"]
Oct 30 22:14:19.702394 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:19.702346    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-bvtk8"]
Oct 30 22:14:20.530042 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:20.529998    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="384f8a66-b2a4-4481-9416-00443b153db9" path="/var/lib/kubelet/pods/384f8a66-b2a4-4481-9416-00443b153db9/volumes"
Oct 30 22:14:24.668640 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:14:24.668602    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 22:14:27.949820 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:27.949784    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-p6nk7"]
Oct 30 22:14:27.950293 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:27.949988    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-p6nk7" podUID="c31e17a7-abf1-4552-8ec5-d9edc8e11314" containerName="istio-proxy" containerID="cri-o://ebe5e3c0b2ca1d3b8517024f2ad6ea626af9efcde6ec6e2d730c09ca0e613434" gracePeriod=30
Oct 30 22:14:27.996555 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:27.996497    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-h8wx4"]
Oct 30 22:14:27.996732 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:27.996713    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-h8wx4" podUID="8467f23a-1b0c-4d55-a1de-efa04241cf30" containerName="istio-proxy" containerID="cri-o://c986b06a87de45bb4e8aab38499e1f8b678031fe7bb60b3c403e00ca54bcaff0" gracePeriod=30
Oct 30 22:14:33.296123 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:33.296095    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-p6nk7"
Oct 30 22:14:33.320786 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:33.320755    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-h8wx4"
Oct 30 22:14:33.387736 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:33.387707    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8467f23a-1b0c-4d55-a1de-efa04241cf30-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8467f23a-1b0c-4d55-a1de-efa04241cf30-istio-envoy\") pod \"8467f23a-1b0c-4d55-a1de-efa04241cf30\" (UID: \"8467f23a-1b0c-4d55-a1de-efa04241cf30\") "
Oct 30 22:14:33.387913 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:33.387756    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8467f23a-1b0c-4d55-a1de-efa04241cf30-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8467f23a-1b0c-4d55-a1de-efa04241cf30-credential-socket\") pod \"8467f23a-1b0c-4d55-a1de-efa04241cf30\" (UID: \"8467f23a-1b0c-4d55-a1de-efa04241cf30\") "
Oct 30 22:14:33.387913 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:33.387779    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c31e17a7-abf1-4552-8ec5-d9edc8e11314-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c31e17a7-abf1-4552-8ec5-d9edc8e11314-istio-data\") pod \"c31e17a7-abf1-4552-8ec5-d9edc8e11314\" (UID: \"c31e17a7-abf1-4552-8ec5-d9edc8e11314\") "
Oct 30 22:14:33.387913 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:33.387804    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8467f23a-1b0c-4d55-a1de-efa04241cf30-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8467f23a-1b0c-4d55-a1de-efa04241cf30-istio-data\") pod \"8467f23a-1b0c-4d55-a1de-efa04241cf30\" (UID: \"8467f23a-1b0c-4d55-a1de-efa04241cf30\") "
Oct 30 22:14:33.387913 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:33.387844    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c31e17a7-abf1-4552-8ec5-d9edc8e11314-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c31e17a7-abf1-4552-8ec5-d9edc8e11314-workload-socket\") pod \"c31e17a7-abf1-4552-8ec5-d9edc8e11314\" (UID: \"c31e17a7-abf1-4552-8ec5-d9edc8e11314\") "
Oct 30 22:14:33.388111 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:33.387919    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8467f23a-1b0c-4d55-a1de-efa04241cf30-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8467f23a-1b0c-4d55-a1de-efa04241cf30-istio-podinfo\") pod \"8467f23a-1b0c-4d55-a1de-efa04241cf30\" (UID: \"8467f23a-1b0c-4d55-a1de-efa04241cf30\") "
Oct 30 22:14:33.388111 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:33.387961    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c31e17a7-abf1-4552-8ec5-d9edc8e11314-kube-api-access-qdmss\" (UniqueName: \"kubernetes.io/projected/c31e17a7-abf1-4552-8ec5-d9edc8e11314-kube-api-access-qdmss\") pod \"c31e17a7-abf1-4552-8ec5-d9edc8e11314\" (UID: \"c31e17a7-abf1-4552-8ec5-d9edc8e11314\") "
Oct 30 22:14:33.388111 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:33.387990    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8467f23a-1b0c-4d55-a1de-efa04241cf30-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8467f23a-1b0c-4d55-a1de-efa04241cf30-istiod-ca-cert\") pod \"8467f23a-1b0c-4d55-a1de-efa04241cf30\" (UID: \"8467f23a-1b0c-4d55-a1de-efa04241cf30\") "
Oct 30 22:14:33.388111 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:33.388037    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c31e17a7-abf1-4552-8ec5-d9edc8e11314-istio-token\" (UniqueName: \"kubernetes.io/projected/c31e17a7-abf1-4552-8ec5-d9edc8e11314-istio-token\") pod \"c31e17a7-abf1-4552-8ec5-d9edc8e11314\" (UID: \"c31e17a7-abf1-4552-8ec5-d9edc8e11314\") "
Oct 30 22:14:33.388340 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:33.388104    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c31e17a7-abf1-4552-8ec5-d9edc8e11314-istio-data" pod "c31e17a7-abf1-4552-8ec5-d9edc8e11314" (UID: "c31e17a7-abf1-4552-8ec5-d9edc8e11314"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:33.388340 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:33.388142    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c31e17a7-abf1-4552-8ec5-d9edc8e11314-workload-socket" pod "c31e17a7-abf1-4552-8ec5-d9edc8e11314" (UID: "c31e17a7-abf1-4552-8ec5-d9edc8e11314"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:33.388340 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:33.388150    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8467f23a-1b0c-4d55-a1de-efa04241cf30-istio-data" pod "8467f23a-1b0c-4d55-a1de-efa04241cf30" (UID: "8467f23a-1b0c-4d55-a1de-efa04241cf30"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:33.388340 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:33.388186    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c31e17a7-abf1-4552-8ec5-d9edc8e11314-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c31e17a7-abf1-4552-8ec5-d9edc8e11314-workload-certs\") pod \"c31e17a7-abf1-4552-8ec5-d9edc8e11314\" (UID: \"c31e17a7-abf1-4552-8ec5-d9edc8e11314\") "
Oct 30 22:14:33.388340 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:33.388230    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c31e17a7-abf1-4552-8ec5-d9edc8e11314-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c31e17a7-abf1-4552-8ec5-d9edc8e11314-istio-envoy\") pod \"c31e17a7-abf1-4552-8ec5-d9edc8e11314\" (UID: \"c31e17a7-abf1-4552-8ec5-d9edc8e11314\") "
Oct 30 22:14:33.388676 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:33.388339    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c31e17a7-abf1-4552-8ec5-d9edc8e11314-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c31e17a7-abf1-4552-8ec5-d9edc8e11314-istiod-ca-cert\") pod \"c31e17a7-abf1-4552-8ec5-d9edc8e11314\" (UID: \"c31e17a7-abf1-4552-8ec5-d9edc8e11314\") "
Oct 30 22:14:33.388676 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:33.388380    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8467f23a-1b0c-4d55-a1de-efa04241cf30-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8467f23a-1b0c-4d55-a1de-efa04241cf30-workload-socket\") pod \"8467f23a-1b0c-4d55-a1de-efa04241cf30\" (UID: \"8467f23a-1b0c-4d55-a1de-efa04241cf30\") "
Oct 30 22:14:33.388676 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:33.388412    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8467f23a-1b0c-4d55-a1de-efa04241cf30-credential-socket" pod "8467f23a-1b0c-4d55-a1de-efa04241cf30" (UID: "8467f23a-1b0c-4d55-a1de-efa04241cf30"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:33.388676 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:33.388447    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c31e17a7-abf1-4552-8ec5-d9edc8e11314-workload-certs" pod "c31e17a7-abf1-4552-8ec5-d9edc8e11314" (UID: "c31e17a7-abf1-4552-8ec5-d9edc8e11314"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:33.388676 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:33.388464    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8467f23a-1b0c-4d55-a1de-efa04241cf30-istio-token\" (UniqueName: \"kubernetes.io/projected/8467f23a-1b0c-4d55-a1de-efa04241cf30-istio-token\") pod \"8467f23a-1b0c-4d55-a1de-efa04241cf30\" (UID: \"8467f23a-1b0c-4d55-a1de-efa04241cf30\") "
Oct 30 22:14:33.388676 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:33.388473    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8467f23a-1b0c-4d55-a1de-efa04241cf30-istiod-ca-cert" pod "8467f23a-1b0c-4d55-a1de-efa04241cf30" (UID: "8467f23a-1b0c-4d55-a1de-efa04241cf30"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:14:33.388676 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:33.388519    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c31e17a7-abf1-4552-8ec5-d9edc8e11314-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c31e17a7-abf1-4552-8ec5-d9edc8e11314-istio-podinfo\") pod \"c31e17a7-abf1-4552-8ec5-d9edc8e11314\" (UID: \"c31e17a7-abf1-4552-8ec5-d9edc8e11314\") "
Oct 30 22:14:33.388676 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:33.388650    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c31e17a7-abf1-4552-8ec5-d9edc8e11314-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c31e17a7-abf1-4552-8ec5-d9edc8e11314-credential-socket\") pod \"c31e17a7-abf1-4552-8ec5-d9edc8e11314\" (UID: \"c31e17a7-abf1-4552-8ec5-d9edc8e11314\") "
Oct 30 22:14:33.389111 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:33.388691    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8467f23a-1b0c-4d55-a1de-efa04241cf30-kube-api-access-t42hk\" (UniqueName: \"kubernetes.io/projected/8467f23a-1b0c-4d55-a1de-efa04241cf30-kube-api-access-t42hk\") pod \"8467f23a-1b0c-4d55-a1de-efa04241cf30\" (UID: \"8467f23a-1b0c-4d55-a1de-efa04241cf30\") "
Oct 30 22:14:33.389111 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:33.388736    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8467f23a-1b0c-4d55-a1de-efa04241cf30-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8467f23a-1b0c-4d55-a1de-efa04241cf30-workload-certs\") pod \"8467f23a-1b0c-4d55-a1de-efa04241cf30\" (UID: \"8467f23a-1b0c-4d55-a1de-efa04241cf30\") "
Oct 30 22:14:33.389111 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:33.388737    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8467f23a-1b0c-4d55-a1de-efa04241cf30-workload-socket" pod "8467f23a-1b0c-4d55-a1de-efa04241cf30" (UID: "8467f23a-1b0c-4d55-a1de-efa04241cf30"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:33.389111 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:33.388808    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c31e17a7-abf1-4552-8ec5-d9edc8e11314-istiod-ca-cert" pod "c31e17a7-abf1-4552-8ec5-d9edc8e11314" (UID: "c31e17a7-abf1-4552-8ec5-d9edc8e11314"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:14:33.389111 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:33.388990    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c31e17a7-abf1-4552-8ec5-d9edc8e11314-credential-socket" pod "c31e17a7-abf1-4552-8ec5-d9edc8e11314" (UID: "c31e17a7-abf1-4552-8ec5-d9edc8e11314"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:33.389393 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:33.389159    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8467f23a-1b0c-4d55-a1de-efa04241cf30-workload-certs" pod "8467f23a-1b0c-4d55-a1de-efa04241cf30" (UID: "8467f23a-1b0c-4d55-a1de-efa04241cf30"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:33.389669 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:33.389622    5061 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8467f23a-1b0c-4d55-a1de-efa04241cf30-workload-certs\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:14:33.389669 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:33.389652    5061 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8467f23a-1b0c-4d55-a1de-efa04241cf30-credential-socket\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:14:33.389669 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:33.389670    5061 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c31e17a7-abf1-4552-8ec5-d9edc8e11314-istio-data\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:14:33.389895 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:33.389685    5061 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8467f23a-1b0c-4d55-a1de-efa04241cf30-istio-data\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:14:33.389895 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:33.389700    5061 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c31e17a7-abf1-4552-8ec5-d9edc8e11314-workload-socket\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:14:33.389895 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:33.389715    5061 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8467f23a-1b0c-4d55-a1de-efa04241cf30-istiod-ca-cert\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:14:33.389895 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:33.389728    5061 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c31e17a7-abf1-4552-8ec5-d9edc8e11314-workload-certs\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:14:33.389895 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:33.389743    5061 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c31e17a7-abf1-4552-8ec5-d9edc8e11314-istiod-ca-cert\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:14:33.389895 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:33.389757    5061 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8467f23a-1b0c-4d55-a1de-efa04241cf30-workload-socket\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:14:33.389895 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:33.389770    5061 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c31e17a7-abf1-4552-8ec5-d9edc8e11314-credential-socket\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:14:33.390916 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:33.390881    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8467f23a-1b0c-4d55-a1de-efa04241cf30-istio-podinfo" pod "8467f23a-1b0c-4d55-a1de-efa04241cf30" (UID: "8467f23a-1b0c-4d55-a1de-efa04241cf30"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:14:33.391048 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:33.391017    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c31e17a7-abf1-4552-8ec5-d9edc8e11314-istio-token" pod "c31e17a7-abf1-4552-8ec5-d9edc8e11314" (UID: "c31e17a7-abf1-4552-8ec5-d9edc8e11314"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:14:33.391333 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:33.391310    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8467f23a-1b0c-4d55-a1de-efa04241cf30-kube-api-access-t42hk" pod "8467f23a-1b0c-4d55-a1de-efa04241cf30" (UID: "8467f23a-1b0c-4d55-a1de-efa04241cf30"). InnerVolumeSpecName "kube-api-access-t42hk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:14:33.391427 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:33.391352    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c31e17a7-abf1-4552-8ec5-d9edc8e11314-kube-api-access-qdmss" pod "c31e17a7-abf1-4552-8ec5-d9edc8e11314" (UID: "c31e17a7-abf1-4552-8ec5-d9edc8e11314"). InnerVolumeSpecName "kube-api-access-qdmss". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:14:33.391427 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:33.391362    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c31e17a7-abf1-4552-8ec5-d9edc8e11314-istio-podinfo" pod "c31e17a7-abf1-4552-8ec5-d9edc8e11314" (UID: "c31e17a7-abf1-4552-8ec5-d9edc8e11314"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:14:33.391427 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:33.391365    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8467f23a-1b0c-4d55-a1de-efa04241cf30-istio-token" pod "8467f23a-1b0c-4d55-a1de-efa04241cf30" (UID: "8467f23a-1b0c-4d55-a1de-efa04241cf30"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:14:33.391626 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:33.391610    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8467f23a-1b0c-4d55-a1de-efa04241cf30-istio-envoy" pod "8467f23a-1b0c-4d55-a1de-efa04241cf30" (UID: "8467f23a-1b0c-4d55-a1de-efa04241cf30"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:33.392464 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:33.392445    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c31e17a7-abf1-4552-8ec5-d9edc8e11314-istio-envoy" pod "c31e17a7-abf1-4552-8ec5-d9edc8e11314" (UID: "c31e17a7-abf1-4552-8ec5-d9edc8e11314"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:33.491588 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:33.491505    5061 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8467f23a-1b0c-4d55-a1de-efa04241cf30-istio-token\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:14:33.491588 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:33.491547    5061 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c31e17a7-abf1-4552-8ec5-d9edc8e11314-istio-podinfo\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:14:33.491588 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:33.491558    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t42hk\" (UniqueName: \"kubernetes.io/projected/8467f23a-1b0c-4d55-a1de-efa04241cf30-kube-api-access-t42hk\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:14:33.491588 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:33.491568    5061 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8467f23a-1b0c-4d55-a1de-efa04241cf30-istio-envoy\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:14:33.491588 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:33.491577    5061 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8467f23a-1b0c-4d55-a1de-efa04241cf30-istio-podinfo\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:14:33.491588 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:33.491587    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdmss\" (UniqueName: \"kubernetes.io/projected/c31e17a7-abf1-4552-8ec5-d9edc8e11314-kube-api-access-qdmss\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:14:33.491923 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:33.491595    5061 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c31e17a7-abf1-4552-8ec5-d9edc8e11314-istio-token\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:14:33.491923 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:33.491604    5061 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c31e17a7-abf1-4552-8ec5-d9edc8e11314-istio-envoy\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:14:33.557833 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:33.557795    5061 generic.go:356] "Generic (PLEG): container finished" podID="8467f23a-1b0c-4d55-a1de-efa04241cf30" containerID="c986b06a87de45bb4e8aab38499e1f8b678031fe7bb60b3c403e00ca54bcaff0" exitCode=0
Oct 30 22:14:33.558061 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:33.558038    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-h8wx4" event={"ID":"8467f23a-1b0c-4d55-a1de-efa04241cf30","Type":"ContainerDied","Data":"c986b06a87de45bb4e8aab38499e1f8b678031fe7bb60b3c403e00ca54bcaff0"}
Oct 30 22:14:33.558150 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:33.558100    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-h8wx4" event={"ID":"8467f23a-1b0c-4d55-a1de-efa04241cf30","Type":"ContainerDied","Data":"74df9466ac92c5c622df6c3b8a8341bbfa2b59c03d33f45e01644e6627039a31"}
Oct 30 22:14:33.558150 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:33.558132    5061 scope.go:122] "RemoveContainer" containerID="c986b06a87de45bb4e8aab38499e1f8b678031fe7bb60b3c403e00ca54bcaff0"
Oct 30 22:14:33.558310 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:33.558294    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-h8wx4"
Oct 30 22:14:33.566824 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:33.566793    5061 generic.go:356] "Generic (PLEG): container finished" podID="c31e17a7-abf1-4552-8ec5-d9edc8e11314" containerID="ebe5e3c0b2ca1d3b8517024f2ad6ea626af9efcde6ec6e2d730c09ca0e613434" exitCode=0
Oct 30 22:14:33.567128 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:33.566976    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-p6nk7" event={"ID":"c31e17a7-abf1-4552-8ec5-d9edc8e11314","Type":"ContainerDied","Data":"ebe5e3c0b2ca1d3b8517024f2ad6ea626af9efcde6ec6e2d730c09ca0e613434"}
Oct 30 22:14:33.567316 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:33.567294    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-p6nk7" event={"ID":"c31e17a7-abf1-4552-8ec5-d9edc8e11314","Type":"ContainerDied","Data":"7ddad95bfafe11329fcb1dfce9548748c5a565bf1326559a026871f5c399a4fc"}
Oct 30 22:14:33.567468 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:33.567058    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-p6nk7"
Oct 30 22:14:33.573867 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:33.573811    5061 scope.go:122] "RemoveContainer" containerID="c986b06a87de45bb4e8aab38499e1f8b678031fe7bb60b3c403e00ca54bcaff0"
Oct 30 22:14:33.574357 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:14:33.574330    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c986b06a87de45bb4e8aab38499e1f8b678031fe7bb60b3c403e00ca54bcaff0\": container with ID starting with c986b06a87de45bb4e8aab38499e1f8b678031fe7bb60b3c403e00ca54bcaff0 not found: ID does not exist" containerID="c986b06a87de45bb4e8aab38499e1f8b678031fe7bb60b3c403e00ca54bcaff0"
Oct 30 22:14:33.574456 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:33.574368    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c986b06a87de45bb4e8aab38499e1f8b678031fe7bb60b3c403e00ca54bcaff0"} err="failed to get container status \"c986b06a87de45bb4e8aab38499e1f8b678031fe7bb60b3c403e00ca54bcaff0\": rpc error: code = NotFound desc = could not find container \"c986b06a87de45bb4e8aab38499e1f8b678031fe7bb60b3c403e00ca54bcaff0\": container with ID starting with c986b06a87de45bb4e8aab38499e1f8b678031fe7bb60b3c403e00ca54bcaff0 not found: ID does not exist"
Oct 30 22:14:33.574456 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:33.574393    5061 scope.go:122] "RemoveContainer" containerID="ebe5e3c0b2ca1d3b8517024f2ad6ea626af9efcde6ec6e2d730c09ca0e613434"
Oct 30 22:14:33.593293 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:33.593262    5061 scope.go:122] "RemoveContainer" containerID="ebe5e3c0b2ca1d3b8517024f2ad6ea626af9efcde6ec6e2d730c09ca0e613434"
Oct 30 22:14:33.593908 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:14:33.593886    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebe5e3c0b2ca1d3b8517024f2ad6ea626af9efcde6ec6e2d730c09ca0e613434\": container with ID starting with ebe5e3c0b2ca1d3b8517024f2ad6ea626af9efcde6ec6e2d730c09ca0e613434 not found: ID does not exist" containerID="ebe5e3c0b2ca1d3b8517024f2ad6ea626af9efcde6ec6e2d730c09ca0e613434"
Oct 30 22:14:33.593995 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:33.593917    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebe5e3c0b2ca1d3b8517024f2ad6ea626af9efcde6ec6e2d730c09ca0e613434"} err="failed to get container status \"ebe5e3c0b2ca1d3b8517024f2ad6ea626af9efcde6ec6e2d730c09ca0e613434\": rpc error: code = NotFound desc = could not find container \"ebe5e3c0b2ca1d3b8517024f2ad6ea626af9efcde6ec6e2d730c09ca0e613434\": container with ID starting with ebe5e3c0b2ca1d3b8517024f2ad6ea626af9efcde6ec6e2d730c09ca0e613434 not found: ID does not exist"
Oct 30 22:14:33.716835 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:33.716802    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-p6nk7"]
Oct 30 22:14:33.763472 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:33.763398    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-p6nk7"]
Oct 30 22:14:33.867739 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:33.867703    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-h8wx4"]
Oct 30 22:14:33.914285 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:33.914252    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-h8wx4"]
Oct 30 22:14:34.529946 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:34.529916    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8467f23a-1b0c-4d55-a1de-efa04241cf30" path="/var/lib/kubelet/pods/8467f23a-1b0c-4d55-a1de-efa04241cf30/volumes"
Oct 30 22:14:34.530465 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:34.530448    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c31e17a7-abf1-4552-8ec5-d9edc8e11314" path="/var/lib/kubelet/pods/c31e17a7-abf1-4552-8ec5-d9edc8e11314/volumes"
Oct 30 22:14:35.335410 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:35.335372    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-c9dc10-576f57cf6c-h5z87"]
Oct 30 22:14:35.336168 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:35.336144    5061 cpu_manager.go:414] "containerMap: removing container" podUID="8467f23a-1b0c-4d55-a1de-efa04241cf30" containerName="istio-proxy"
Oct 30 22:14:35.336168 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:35.336166    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8467f23a-1b0c-4d55-a1de-efa04241cf30" containerName="istio-proxy"
Oct 30 22:14:35.336340 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:35.336196    5061 cpu_manager.go:414] "containerMap: removing container" podUID="c31e17a7-abf1-4552-8ec5-d9edc8e11314" containerName="istio-proxy"
Oct 30 22:14:35.336340 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:35.336201    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c31e17a7-abf1-4552-8ec5-d9edc8e11314" containerName="istio-proxy"
Oct 30 22:14:35.336340 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:35.336225    5061 cpu_manager.go:414] "containerMap: removing container" podUID="384f8a66-b2a4-4481-9416-00443b153db9" containerName="istio-proxy"
Oct 30 22:14:35.336340 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:35.336230    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="384f8a66-b2a4-4481-9416-00443b153db9" containerName="istio-proxy"
Oct 30 22:14:35.336475 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:35.336391    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="8467f23a-1b0c-4d55-a1de-efa04241cf30" containerName="istio-proxy"
Oct 30 22:14:35.336475 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:35.336401    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="c31e17a7-abf1-4552-8ec5-d9edc8e11314" containerName="istio-proxy"
Oct 30 22:14:35.336475 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:35.336414    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="384f8a66-b2a4-4481-9416-00443b153db9" containerName="istio-proxy"
Oct 30 22:14:35.341898 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:35.341879    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-c9dc10-576f57cf6c-h5z87"
Oct 30 22:14:35.421210 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:35.421176    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7l6nl\" (UniqueName: \"kubernetes.io/projected/db0dfe9a-a006-41e5-8d87-72eb15fae883-kube-api-access-7l6nl\") pod \"maas-api-e2e-shared-b-w6-c9dc10-576f57cf6c-h5z87\" (UID: \"db0dfe9a-a006-41e5-8d87-72eb15fae883\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-c9dc10-576f57cf6c-h5z87"
Oct 30 22:14:35.421374 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:35.421323    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db0dfe9a-a006-41e5-8d87-72eb15fae883-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-c9dc10-576f57cf6c-h5z87\" (UID: \"db0dfe9a-a006-41e5-8d87-72eb15fae883\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-c9dc10-576f57cf6c-h5z87"
Oct 30 22:14:35.421374 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:35.421364    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db0dfe9a-a006-41e5-8d87-72eb15fae883-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-c9dc10-576f57cf6c-h5z87\" (UID: \"db0dfe9a-a006-41e5-8d87-72eb15fae883\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-c9dc10-576f57cf6c-h5z87"
Oct 30 22:14:35.421451 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:35.421407    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db0dfe9a-a006-41e5-8d87-72eb15fae883-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-c9dc10-576f57cf6c-h5z87\" (UID: \"db0dfe9a-a006-41e5-8d87-72eb15fae883\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-c9dc10-576f57cf6c-h5z87"
Oct 30 22:14:35.439539 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:35.439500    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w6-c9dc10-serving-cert\""
Oct 30 22:14:35.444832 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:35.444809    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-c9dc10-576f57cf6c-h5z87"]
Oct 30 22:14:35.523409 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:35.523368    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7l6nl\" (UniqueName: \"kubernetes.io/projected/db0dfe9a-a006-41e5-8d87-72eb15fae883-kube-api-access-7l6nl\") pod \"maas-api-e2e-shared-b-w6-c9dc10-576f57cf6c-h5z87\" (UID: \"db0dfe9a-a006-41e5-8d87-72eb15fae883\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-c9dc10-576f57cf6c-h5z87"
Oct 30 22:14:35.523615 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:35.523483    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db0dfe9a-a006-41e5-8d87-72eb15fae883-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-c9dc10-576f57cf6c-h5z87\" (UID: \"db0dfe9a-a006-41e5-8d87-72eb15fae883\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-c9dc10-576f57cf6c-h5z87"
Oct 30 22:14:35.523615 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:35.523549    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db0dfe9a-a006-41e5-8d87-72eb15fae883-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-c9dc10-576f57cf6c-h5z87\" (UID: \"db0dfe9a-a006-41e5-8d87-72eb15fae883\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-c9dc10-576f57cf6c-h5z87"
Oct 30 22:14:35.523615 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:35.523601    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db0dfe9a-a006-41e5-8d87-72eb15fae883-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-c9dc10-576f57cf6c-h5z87\" (UID: \"db0dfe9a-a006-41e5-8d87-72eb15fae883\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-c9dc10-576f57cf6c-h5z87"
Oct 30 22:14:35.523792 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:14:35.523659    5061 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-c9dc10-serving-cert: secret "maas-api-e2e-shared-b-w6-c9dc10-serving-cert" not found
Oct 30 22:14:35.523792 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:14:35.523720    5061 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db0dfe9a-a006-41e5-8d87-72eb15fae883-maas-api-tls podName:db0dfe9a-a006-41e5-8d87-72eb15fae883 nodeName:}" failed. No retries permitted until 2026-10-30 22:14:36.023702993 +0000 UTC m=+1929.636472757 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/db0dfe9a-a006-41e5-8d87-72eb15fae883-maas-api-tls") pod "maas-api-e2e-shared-b-w6-c9dc10-576f57cf6c-h5z87" (UID: "db0dfe9a-a006-41e5-8d87-72eb15fae883") : secret "maas-api-e2e-shared-b-w6-c9dc10-serving-cert" not found
Oct 30 22:14:35.524251 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:35.524233    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db0dfe9a-a006-41e5-8d87-72eb15fae883-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-c9dc10-576f57cf6c-h5z87\" (UID: \"db0dfe9a-a006-41e5-8d87-72eb15fae883\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-c9dc10-576f57cf6c-h5z87"
Oct 30 22:14:35.526139 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:35.526120    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db0dfe9a-a006-41e5-8d87-72eb15fae883-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-c9dc10-576f57cf6c-h5z87\" (UID: \"db0dfe9a-a006-41e5-8d87-72eb15fae883\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-c9dc10-576f57cf6c-h5z87"
Oct 30 22:14:35.606076 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:35.604627    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7l6nl\" (UniqueName: \"kubernetes.io/projected/db0dfe9a-a006-41e5-8d87-72eb15fae883-kube-api-access-7l6nl\") pod \"maas-api-e2e-shared-b-w6-c9dc10-576f57cf6c-h5z87\" (UID: \"db0dfe9a-a006-41e5-8d87-72eb15fae883\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-c9dc10-576f57cf6c-h5z87"
Oct 30 22:14:36.032841 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:36.032751    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db0dfe9a-a006-41e5-8d87-72eb15fae883-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-c9dc10-576f57cf6c-h5z87\" (UID: \"db0dfe9a-a006-41e5-8d87-72eb15fae883\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-c9dc10-576f57cf6c-h5z87"
Oct 30 22:14:36.035222 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:36.035201    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db0dfe9a-a006-41e5-8d87-72eb15fae883-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-c9dc10-576f57cf6c-h5z87\" (UID: \"db0dfe9a-a006-41e5-8d87-72eb15fae883\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-c9dc10-576f57cf6c-h5z87"
Oct 30 22:14:36.256668 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:36.256636    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-c9dc10-576f57cf6c-h5z87"
Oct 30 22:14:36.633144 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:36.631210    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-c9dc10-576f57cf6c-h5z87"]
Oct 30 22:14:36.633144 ip-10-0-130-60 kubenswrapper[5061]: W1030 22:14:36.631674    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb0dfe9a_a006_41e5_8d87_72eb15fae883.slice/crio-c7bbae67aa728a1ea44cfe5dd7da481bf4a62ee42e425be5bc717a9f79eec85a WatchSource:0}: Error finding container c7bbae67aa728a1ea44cfe5dd7da481bf4a62ee42e425be5bc717a9f79eec85a: Status 404 returned error can't find the container with id c7bbae67aa728a1ea44cfe5dd7da481bf4a62ee42e425be5bc717a9f79eec85a
Oct 30 22:14:37.640905 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:37.640863    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-c9dc10-576f57cf6c-h5z87" event={"ID":"db0dfe9a-a006-41e5-8d87-72eb15fae883","Type":"ContainerStarted","Data":"c7bbae67aa728a1ea44cfe5dd7da481bf4a62ee42e425be5bc717a9f79eec85a"}
Oct 30 22:14:38.650046 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:38.649966    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-c9dc10-576f57cf6c-h5z87" event={"ID":"db0dfe9a-a006-41e5-8d87-72eb15fae883","Type":"ContainerStarted","Data":"48f23f17e79d55618fa4c03db7cf0097db5466c8e3fdfacc8cd92c111466daee"}
Oct 30 22:14:38.650046 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:38.650032    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-c9dc10-576f57cf6c-h5z87"
Oct 30 22:14:38.807597 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:38.807495    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-c9dc10-576f57cf6c-h5z87" podStartSLOduration=2.871939703 podStartE2EDuration="3.807477546s" totalImagesPullingTime="935.537843ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:14:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:14:38.80461559 +0000 UTC m=+1932.417385349" watchObservedRunningTime="2026-10-30 22:14:38.807477546 +0000 UTC m=+1932.420247306"
Oct 30 22:14:44.663372 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:14:44.663337    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-c9dc10-576f57cf6c-h5z87"
Oct 30 22:14:54.668480 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:14:54.668442    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 22:15:00.192695 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:00.192662    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-6ea8da-29889975-r8jsf"]
Oct 30 22:15:00.197132 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:00.197113    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-6ea8da-29889975-r8jsf"
Oct 30 22:15:00.207373 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:00.207354    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-4d8041-29889975-fn5gn"]
Oct 30 22:15:00.211997 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:00.211976    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-02bd3b-29889975-jjfj2"]
Oct 30 22:15:00.212099 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:00.212078    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-4d8041-29889975-fn5gn"
Oct 30 22:15:00.216238 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:00.216217    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-02bd3b-29889975-jjfj2"
Oct 30 22:15:00.239344 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:00.239319    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-778d39d3-29889975-mct28"]
Oct 30 22:15:00.243445 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:00.243426    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-411ff1-29889975-fc8l4"]
Oct 30 22:15:00.243585 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:00.243567    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-778d39d3-29889975-mct28"
Oct 30 22:15:00.247672 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:00.247655    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-411ff1-29889975-fc8l4"
Oct 30 22:15:00.250345 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:00.250302    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gdmfg\""
Oct 30 22:15:00.260731 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:00.260711    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5402dd-29889975-jjcmb"]
Oct 30 22:15:00.265134 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:00.265115    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5402dd-29889975-jjcmb"
Oct 30 22:15:00.286504 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:00.286479    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889975-dq9xr"]
Oct 30 22:15:00.290715 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:00.290693    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-6ea8da-29889975-r8jsf"]
Oct 30 22:15:00.290810 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:00.290701    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-dq9xr"
Oct 30 22:15:00.301088 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:00.301070    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-4d8041-29889975-fn5gn"]
Oct 30 22:15:00.302678 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:00.302659    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08aac4a2-7d84-4b74-bf47-dc849fd6e446-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-6ea8da-29889975-r8jsf\" (UID: \"08aac4a2-7d84-4b74-bf47-dc849fd6e446\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-6ea8da-29889975-r8jsf"
Oct 30 22:15:00.302866 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:00.302846    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngvv2\" (UniqueName: \"kubernetes.io/projected/08aac4a2-7d84-4b74-bf47-dc849fd6e446-kube-api-access-ngvv2\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-6ea8da-29889975-r8jsf\" (UID: \"08aac4a2-7d84-4b74-bf47-dc849fd6e446\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-6ea8da-29889975-r8jsf"
Oct 30 22:15:00.305354 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:00.305333    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-25de17-29889975-qk5j5"]
Oct 30 22:15:00.309397 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:00.309378    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-25de17-29889975-qk5j5"
Oct 30 22:15:00.331183 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:00.330055    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5fef81-29889975-fdwmx"]
Oct 30 22:15:00.335643 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:00.335618    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5fef81-29889975-fdwmx"
Oct 30 22:15:00.343585 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:00.343506    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-02bd3b-29889975-jjfj2"]
Oct 30 22:15:00.353616 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:00.353593    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5402dd-29889975-jjcmb"]
Oct 30 22:15:00.358757 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:00.358734    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-778d39d3-29889975-mct28"]
Oct 30 22:15:00.375401 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:00.375098    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e66edb-29889975-flzf6"]
Oct 30 22:15:00.381365 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:00.381342    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e66edb-29889975-flzf6"
Oct 30 22:15:00.385451 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:00.385211    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889975-dq9xr"]
Oct 30 22:15:00.390139 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:00.389972    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-25de17-29889975-qk5j5"]
Oct 30 22:15:00.403962 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:00.403937    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42adc96b-1b21-484e-b6fa-51bbf1127ac0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-4d8041-29889975-fn5gn\" (UID: \"42adc96b-1b21-484e-b6fa-51bbf1127ac0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-4d8041-29889975-fn5gn"
Oct 30 22:15:00.404093 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:00.403969    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ba41c1f-1fc3-4a32-967c-2d86878e2887-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-5402dd-29889975-jjcmb\" (UID: \"0ba41c1f-1fc3-4a32-967c-2d86878e2887\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5402dd-29889975-jjcmb"
Oct 30 22:15:00.404093 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:00.404061    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llp8p\" (UniqueName: \"kubernetes.io/projected/3787aeb2-4e6e-4855-8be8-137ebcd39a26-kube-api-access-llp8p\") pod \"maas-api-key-cleanup-e2e-mt-778d39d3-29889975-mct28\" (UID: \"3787aeb2-4e6e-4855-8be8-137ebcd39a26\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-778d39d3-29889975-mct28"
Oct 30 22:15:00.404221 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:00.404122    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08aac4a2-7d84-4b74-bf47-dc849fd6e446-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-6ea8da-29889975-r8jsf\" (UID: \"08aac4a2-7d84-4b74-bf47-dc849fd6e446\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-6ea8da-29889975-r8jsf"
Oct 30 22:15:00.404221 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:00.404145    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f220141f-4150-4f9c-9e14-631777d0ac14-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889975-dq9xr\" (UID: \"f220141f-4150-4f9c-9e14-631777d0ac14\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-dq9xr"
Oct 30 22:15:00.404221 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:00.404184    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcbck\" (UniqueName: \"kubernetes.io/projected/0ba41c1f-1fc3-4a32-967c-2d86878e2887-kube-api-access-mcbck\") pod \"maas-api-key-cleanup-e2e-worker-w5-5402dd-29889975-jjcmb\" (UID: \"0ba41c1f-1fc3-4a32-967c-2d86878e2887\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5402dd-29889975-jjcmb"
Oct 30 22:15:00.404391 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:00.404230    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwjn9\" (UniqueName: \"kubernetes.io/projected/f220141f-4150-4f9c-9e14-631777d0ac14-kube-api-access-dwjn9\") pod \"maas-api-key-cleanup-29889975-dq9xr\" (UID: \"f220141f-4150-4f9c-9e14-631777d0ac14\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-dq9xr"
Oct 30 22:15:00.404391 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:00.404256    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52312d9d-e2f8-4ad3-bce1-53ffc5b08d4f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-411ff1-29889975-fc8l4\" (UID: \"52312d9d-e2f8-4ad3-bce1-53ffc5b08d4f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-411ff1-29889975-fc8l4"
Oct 30 22:15:00.404391 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:00.404276    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfkks\" (UniqueName: \"kubernetes.io/projected/52312d9d-e2f8-4ad3-bce1-53ffc5b08d4f-kube-api-access-lfkks\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-411ff1-29889975-fc8l4\" (UID: \"52312d9d-e2f8-4ad3-bce1-53ffc5b08d4f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-411ff1-29889975-fc8l4"
Oct 30 22:15:00.404391 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:00.404299    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3787aeb2-4e6e-4855-8be8-137ebcd39a26-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-778d39d3-29889975-mct28\" (UID: \"3787aeb2-4e6e-4855-8be8-137ebcd39a26\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-778d39d3-29889975-mct28"
Oct 30 22:15:00.404870 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:00.404830    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nb7jr\" (UniqueName: \"kubernetes.io/projected/06e58c0b-aa5c-4f1e-8228-7070a03657bb-kube-api-access-nb7jr\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-02bd3b-29889975-jjfj2\" (UID: \"06e58c0b-aa5c-4f1e-8228-7070a03657bb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-02bd3b-29889975-jjfj2"
Oct 30 22:15:00.404975 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:00.404955    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gf2fw\" (UniqueName: \"kubernetes.io/projected/42adc96b-1b21-484e-b6fa-51bbf1127ac0-kube-api-access-gf2fw\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-4d8041-29889975-fn5gn\" (UID: \"42adc96b-1b21-484e-b6fa-51bbf1127ac0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-4d8041-29889975-fn5gn"
Oct 30 22:15:00.405048 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:00.405008    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06e58c0b-aa5c-4f1e-8228-7070a03657bb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-02bd3b-29889975-jjfj2\" (UID: \"06e58c0b-aa5c-4f1e-8228-7070a03657bb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-02bd3b-29889975-jjfj2"
Oct 30 22:15:00.405112 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:00.405044    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngvv2\" (UniqueName: \"kubernetes.io/projected/08aac4a2-7d84-4b74-bf47-dc849fd6e446-kube-api-access-ngvv2\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-6ea8da-29889975-r8jsf\" (UID: \"08aac4a2-7d84-4b74-bf47-dc849fd6e446\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-6ea8da-29889975-r8jsf"
Oct 30 22:15:00.405167 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:00.405107    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08aac4a2-7d84-4b74-bf47-dc849fd6e446-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-6ea8da-29889975-r8jsf\" (UID: \"08aac4a2-7d84-4b74-bf47-dc849fd6e446\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-6ea8da-29889975-r8jsf"
Oct 30 22:15:00.432031 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:00.431992    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5fef81-29889975-fdwmx"]
Oct 30 22:15:00.433833 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:00.433323    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-411ff1-29889975-fc8l4"]
Oct 30 22:15:00.447855 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:00.447776    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e66edb-29889975-flzf6"]
Oct 30 22:15:00.460624 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:00.460593    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngvv2\" (UniqueName: \"kubernetes.io/projected/08aac4a2-7d84-4b74-bf47-dc849fd6e446-kube-api-access-ngvv2\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-6ea8da-29889975-r8jsf\" (UID: \"08aac4a2-7d84-4b74-bf47-dc849fd6e446\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-6ea8da-29889975-r8jsf"
Oct 30 22:15:00.506081 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:00.506046    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nb7jr\" (UniqueName: \"kubernetes.io/projected/06e58c0b-aa5c-4f1e-8228-7070a03657bb-kube-api-access-nb7jr\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-02bd3b-29889975-jjfj2\" (UID: \"06e58c0b-aa5c-4f1e-8228-7070a03657bb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-02bd3b-29889975-jjfj2"
Oct 30 22:15:00.506238 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:00.506091    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gf2fw\" (UniqueName: \"kubernetes.io/projected/42adc96b-1b21-484e-b6fa-51bbf1127ac0-kube-api-access-gf2fw\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-4d8041-29889975-fn5gn\" (UID: \"42adc96b-1b21-484e-b6fa-51bbf1127ac0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-4d8041-29889975-fn5gn"
Oct 30 22:15:00.506238 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:00.506137    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06e58c0b-aa5c-4f1e-8228-7070a03657bb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-02bd3b-29889975-jjfj2\" (UID: \"06e58c0b-aa5c-4f1e-8228-7070a03657bb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-02bd3b-29889975-jjfj2"
Oct 30 22:15:00.506238 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:00.506217    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42adc96b-1b21-484e-b6fa-51bbf1127ac0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-4d8041-29889975-fn5gn\" (UID: \"42adc96b-1b21-484e-b6fa-51bbf1127ac0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-4d8041-29889975-fn5gn"
Oct 30 22:15:00.506238 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:00.506234    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ba41c1f-1fc3-4a32-967c-2d86878e2887-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-5402dd-29889975-jjcmb\" (UID: \"0ba41c1f-1fc3-4a32-967c-2d86878e2887\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5402dd-29889975-jjcmb"
Oct 30 22:15:00.506937 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:00.506311    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llp8p\" (UniqueName: \"kubernetes.io/projected/3787aeb2-4e6e-4855-8be8-137ebcd39a26-kube-api-access-llp8p\") pod \"maas-api-key-cleanup-e2e-mt-778d39d3-29889975-mct28\" (UID: \"3787aeb2-4e6e-4855-8be8-137ebcd39a26\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-778d39d3-29889975-mct28"
Oct 30 22:15:00.507064 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:00.507044    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f220141f-4150-4f9c-9e14-631777d0ac14-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889975-dq9xr\" (UID: \"f220141f-4150-4f9c-9e14-631777d0ac14\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-dq9xr"
Oct 30 22:15:00.507138 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:00.507082    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42adc96b-1b21-484e-b6fa-51bbf1127ac0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-4d8041-29889975-fn5gn\" (UID: \"42adc96b-1b21-484e-b6fa-51bbf1127ac0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-4d8041-29889975-fn5gn"
Oct 30 22:15:00.507138 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:00.507095    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06e58c0b-aa5c-4f1e-8228-7070a03657bb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-02bd3b-29889975-jjfj2\" (UID: \"06e58c0b-aa5c-4f1e-8228-7070a03657bb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-02bd3b-29889975-jjfj2"
Oct 30 22:15:00.507138 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:00.507082    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ba41c1f-1fc3-4a32-967c-2d86878e2887-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-5402dd-29889975-jjcmb\" (UID: \"0ba41c1f-1fc3-4a32-967c-2d86878e2887\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5402dd-29889975-jjcmb"
Oct 30 22:15:00.507138 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:00.507085    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsb6m\" (UniqueName: \"kubernetes.io/projected/cbaff831-8881-4762-8332-25c8eb4c2a28-kube-api-access-qsb6m\") pod \"maas-api-key-cleanup-e2e-api-a-5fef81-29889975-fdwmx\" (UID: \"cbaff831-8881-4762-8332-25c8eb4c2a28\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5fef81-29889975-fdwmx"
Oct 30 22:15:00.507294 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:00.507278    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b4f58e4-2611-45f1-b431-a33ce392d964-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-e66edb-29889975-flzf6\" (UID: \"8b4f58e4-2611-45f1-b431-a33ce392d964\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e66edb-29889975-flzf6"
Oct 30 22:15:00.507363 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:00.507344    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcbck\" (UniqueName: \"kubernetes.io/projected/0ba41c1f-1fc3-4a32-967c-2d86878e2887-kube-api-access-mcbck\") pod \"maas-api-key-cleanup-e2e-worker-w5-5402dd-29889975-jjcmb\" (UID: \"0ba41c1f-1fc3-4a32-967c-2d86878e2887\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5402dd-29889975-jjcmb"
Oct 30 22:15:00.507406 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:00.507379    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/777f3189-c018-4cdb-92ba-eeb10b71e868-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-25de17-29889975-qk5j5\" (UID: \"777f3189-c018-4cdb-92ba-eeb10b71e868\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-25de17-29889975-qk5j5"
Oct 30 22:15:00.507446 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:00.507404    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6cw8\" (UniqueName: \"kubernetes.io/projected/777f3189-c018-4cdb-92ba-eeb10b71e868-kube-api-access-j6cw8\") pod \"maas-api-key-cleanup-e2e-api-a-25de17-29889975-qk5j5\" (UID: \"777f3189-c018-4cdb-92ba-eeb10b71e868\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-25de17-29889975-qk5j5"
Oct 30 22:15:00.507506 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:00.507490    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbaff831-8881-4762-8332-25c8eb4c2a28-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-5fef81-29889975-fdwmx\" (UID: \"cbaff831-8881-4762-8332-25c8eb4c2a28\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5fef81-29889975-fdwmx"
Oct 30 22:15:00.507603 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:00.507585    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwjn9\" (UniqueName: \"kubernetes.io/projected/f220141f-4150-4f9c-9e14-631777d0ac14-kube-api-access-dwjn9\") pod \"maas-api-key-cleanup-29889975-dq9xr\" (UID: \"f220141f-4150-4f9c-9e14-631777d0ac14\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-dq9xr"
Oct 30 22:15:00.507672 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:00.507656    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f220141f-4150-4f9c-9e14-631777d0ac14-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889975-dq9xr\" (UID: \"f220141f-4150-4f9c-9e14-631777d0ac14\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-dq9xr"
Oct 30 22:15:00.507756 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:00.507739    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52312d9d-e2f8-4ad3-bce1-53ffc5b08d4f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-411ff1-29889975-fc8l4\" (UID: \"52312d9d-e2f8-4ad3-bce1-53ffc5b08d4f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-411ff1-29889975-fc8l4"
Oct 30 22:15:00.507822 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:00.507781    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfkks\" (UniqueName: \"kubernetes.io/projected/52312d9d-e2f8-4ad3-bce1-53ffc5b08d4f-kube-api-access-lfkks\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-411ff1-29889975-fc8l4\" (UID: \"52312d9d-e2f8-4ad3-bce1-53ffc5b08d4f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-411ff1-29889975-fc8l4"
Oct 30 22:15:00.507895 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:00.507829    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3787aeb2-4e6e-4855-8be8-137ebcd39a26-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-778d39d3-29889975-mct28\" (UID: \"3787aeb2-4e6e-4855-8be8-137ebcd39a26\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-778d39d3-29889975-mct28"
Oct 30 22:15:00.507895 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:00.507860    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbf6h\" (UniqueName: \"kubernetes.io/projected/8b4f58e4-2611-45f1-b431-a33ce392d964-kube-api-access-gbf6h\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-e66edb-29889975-flzf6\" (UID: \"8b4f58e4-2611-45f1-b431-a33ce392d964\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e66edb-29889975-flzf6"
Oct 30 22:15:00.508321 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:00.508301    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52312d9d-e2f8-4ad3-bce1-53ffc5b08d4f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-411ff1-29889975-fc8l4\" (UID: \"52312d9d-e2f8-4ad3-bce1-53ffc5b08d4f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-411ff1-29889975-fc8l4"
Oct 30 22:15:00.508416 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:00.508398    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3787aeb2-4e6e-4855-8be8-137ebcd39a26-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-778d39d3-29889975-mct28\" (UID: \"3787aeb2-4e6e-4855-8be8-137ebcd39a26\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-778d39d3-29889975-mct28"
Oct 30 22:15:00.512268 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:00.512248    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-6ea8da-29889975-r8jsf"
Oct 30 22:15:00.563235 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:00.563203    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nb7jr\" (UniqueName: \"kubernetes.io/projected/06e58c0b-aa5c-4f1e-8228-7070a03657bb-kube-api-access-nb7jr\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-02bd3b-29889975-jjfj2\" (UID: \"06e58c0b-aa5c-4f1e-8228-7070a03657bb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-02bd3b-29889975-jjfj2"
Oct 30 22:15:00.566121 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:00.566095    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwjn9\" (UniqueName: \"kubernetes.io/projected/f220141f-4150-4f9c-9e14-631777d0ac14-kube-api-access-dwjn9\") pod \"maas-api-key-cleanup-29889975-dq9xr\" (UID: \"f220141f-4150-4f9c-9e14-631777d0ac14\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-dq9xr"
Oct 30 22:15:00.567325 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:00.567295    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gf2fw\" (UniqueName: \"kubernetes.io/projected/42adc96b-1b21-484e-b6fa-51bbf1127ac0-kube-api-access-gf2fw\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-4d8041-29889975-fn5gn\" (UID: \"42adc96b-1b21-484e-b6fa-51bbf1127ac0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-4d8041-29889975-fn5gn"
Oct 30 22:15:00.569855 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:00.569835    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-llp8p\" (UniqueName: \"kubernetes.io/projected/3787aeb2-4e6e-4855-8be8-137ebcd39a26-kube-api-access-llp8p\") pod \"maas-api-key-cleanup-e2e-mt-778d39d3-29889975-mct28\" (UID: \"3787aeb2-4e6e-4855-8be8-137ebcd39a26\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-778d39d3-29889975-mct28"
Oct 30 22:15:00.570267 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:00.570250    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfkks\" (UniqueName: \"kubernetes.io/projected/52312d9d-e2f8-4ad3-bce1-53ffc5b08d4f-kube-api-access-lfkks\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-411ff1-29889975-fc8l4\" (UID: \"52312d9d-e2f8-4ad3-bce1-53ffc5b08d4f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-411ff1-29889975-fc8l4"
Oct 30 22:15:00.570789 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:00.570495    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcbck\" (UniqueName: \"kubernetes.io/projected/0ba41c1f-1fc3-4a32-967c-2d86878e2887-kube-api-access-mcbck\") pod \"maas-api-key-cleanup-e2e-worker-w5-5402dd-29889975-jjcmb\" (UID: \"0ba41c1f-1fc3-4a32-967c-2d86878e2887\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5402dd-29889975-jjcmb"
Oct 30 22:15:00.579863 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:00.579839    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5402dd-29889975-jjcmb"
Oct 30 22:15:00.608465 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:00.608428    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-dq9xr"
Oct 30 22:15:00.609318 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:00.609291    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsb6m\" (UniqueName: \"kubernetes.io/projected/cbaff831-8881-4762-8332-25c8eb4c2a28-kube-api-access-qsb6m\") pod \"maas-api-key-cleanup-e2e-api-a-5fef81-29889975-fdwmx\" (UID: \"cbaff831-8881-4762-8332-25c8eb4c2a28\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5fef81-29889975-fdwmx"
Oct 30 22:15:00.609430 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:00.609380    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b4f58e4-2611-45f1-b431-a33ce392d964-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-e66edb-29889975-flzf6\" (UID: \"8b4f58e4-2611-45f1-b431-a33ce392d964\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e66edb-29889975-flzf6"
Oct 30 22:15:00.609495 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:00.609430    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/777f3189-c018-4cdb-92ba-eeb10b71e868-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-25de17-29889975-qk5j5\" (UID: \"777f3189-c018-4cdb-92ba-eeb10b71e868\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-25de17-29889975-qk5j5"
Oct 30 22:15:00.609495 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:00.609455    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6cw8\" (UniqueName: \"kubernetes.io/projected/777f3189-c018-4cdb-92ba-eeb10b71e868-kube-api-access-j6cw8\") pod \"maas-api-key-cleanup-e2e-api-a-25de17-29889975-qk5j5\" (UID: \"777f3189-c018-4cdb-92ba-eeb10b71e868\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-25de17-29889975-qk5j5"
Oct 30 22:15:00.609651 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:00.609498    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbaff831-8881-4762-8332-25c8eb4c2a28-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-5fef81-29889975-fdwmx\" (UID: \"cbaff831-8881-4762-8332-25c8eb4c2a28\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5fef81-29889975-fdwmx"
Oct 30 22:15:00.611587 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:00.610738    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbf6h\" (UniqueName: \"kubernetes.io/projected/8b4f58e4-2611-45f1-b431-a33ce392d964-kube-api-access-gbf6h\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-e66edb-29889975-flzf6\" (UID: \"8b4f58e4-2611-45f1-b431-a33ce392d964\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e66edb-29889975-flzf6"
Oct 30 22:15:00.611587 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:00.610947    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/777f3189-c018-4cdb-92ba-eeb10b71e868-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-25de17-29889975-qk5j5\" (UID: \"777f3189-c018-4cdb-92ba-eeb10b71e868\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-25de17-29889975-qk5j5"
Oct 30 22:15:00.612755 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:00.612730    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbaff831-8881-4762-8332-25c8eb4c2a28-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-5fef81-29889975-fdwmx\" (UID: \"cbaff831-8881-4762-8332-25c8eb4c2a28\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5fef81-29889975-fdwmx"
Oct 30 22:15:00.612869 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:00.612789    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b4f58e4-2611-45f1-b431-a33ce392d964-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-e66edb-29889975-flzf6\" (UID: \"8b4f58e4-2611-45f1-b431-a33ce392d964\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e66edb-29889975-flzf6"
Oct 30 22:15:00.669376 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:00.669316    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbf6h\" (UniqueName: \"kubernetes.io/projected/8b4f58e4-2611-45f1-b431-a33ce392d964-kube-api-access-gbf6h\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-e66edb-29889975-flzf6\" (UID: \"8b4f58e4-2611-45f1-b431-a33ce392d964\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e66edb-29889975-flzf6"
Oct 30 22:15:00.670166 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:00.670115    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6cw8\" (UniqueName: \"kubernetes.io/projected/777f3189-c018-4cdb-92ba-eeb10b71e868-kube-api-access-j6cw8\") pod \"maas-api-key-cleanup-e2e-api-a-25de17-29889975-qk5j5\" (UID: \"777f3189-c018-4cdb-92ba-eeb10b71e868\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-25de17-29889975-qk5j5"
Oct 30 22:15:00.672065 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:00.672037    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsb6m\" (UniqueName: \"kubernetes.io/projected/cbaff831-8881-4762-8332-25c8eb4c2a28-kube-api-access-qsb6m\") pod \"maas-api-key-cleanup-e2e-api-a-5fef81-29889975-fdwmx\" (UID: \"cbaff831-8881-4762-8332-25c8eb4c2a28\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5fef81-29889975-fdwmx"
Oct 30 22:15:00.703578 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:00.703462    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e66edb-29889975-flzf6"
Oct 30 22:15:00.827608 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:00.827580    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-4d8041-29889975-fn5gn"
Oct 30 22:15:00.834700 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:00.834672    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-02bd3b-29889975-jjfj2"
Oct 30 22:15:00.858247 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:00.858221    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-778d39d3-29889975-mct28"
Oct 30 22:15:00.864857 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:00.864829    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-411ff1-29889975-fc8l4"
Oct 30 22:15:00.903326 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:00.902700    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-6ea8da-29889975-r8jsf"]
Oct 30 22:15:00.925819 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:00.924992    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-25de17-29889975-qk5j5"
Oct 30 22:15:00.965569 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:00.965500    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5fef81-29889975-fdwmx"
Oct 30 22:15:01.011668 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:01.011636    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5402dd-29889975-jjcmb"]
Oct 30 22:15:01.046720 ip-10-0-130-60 kubenswrapper[5061]: W1030 22:15:01.044970    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ba41c1f_1fc3_4a32_967c_2d86878e2887.slice/crio-a4f7142dc6a02edfb7a68046d38c0650e316c7a877c46d032784c3f5810f1c61 WatchSource:0}: Error finding container a4f7142dc6a02edfb7a68046d38c0650e316c7a877c46d032784c3f5810f1c61: Status 404 returned error can't find the container with id a4f7142dc6a02edfb7a68046d38c0650e316c7a877c46d032784c3f5810f1c61
Oct 30 22:15:01.101914 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:01.101879    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e66edb-29889975-flzf6"]
Oct 30 22:15:01.163735 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:01.163702    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889975-dq9xr"]
Oct 30 22:15:01.212591 ip-10-0-130-60 kubenswrapper[5061]: W1030 22:15:01.211373    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf220141f_4150_4f9c_9e14_631777d0ac14.slice/crio-74ea822855504dd1f4fc36a274b5be614dcac462259ee2582463d739c242d447 WatchSource:0}: Error finding container 74ea822855504dd1f4fc36a274b5be614dcac462259ee2582463d739c242d447: Status 404 returned error can't find the container with id 74ea822855504dd1f4fc36a274b5be614dcac462259ee2582463d739c242d447
Oct 30 22:15:01.218113 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:01.218085    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-4d8041-29889975-fn5gn"]
Oct 30 22:15:01.245972 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:01.245934    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-02bd3b-29889975-jjfj2"]
Oct 30 22:15:01.304993 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:01.304966    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-778d39d3-29889975-mct28"]
Oct 30 22:15:01.312057 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:01.311891    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-411ff1-29889975-fc8l4"]
Oct 30 22:15:01.313774 ip-10-0-130-60 kubenswrapper[5061]: W1030 22:15:01.313745    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06e58c0b_aa5c_4f1e_8228_7070a03657bb.slice/crio-72966f1dbddeb6aa1d9643bd82e3aef651da682cb5b9233644ee3e1579518fe9 WatchSource:0}: Error finding container 72966f1dbddeb6aa1d9643bd82e3aef651da682cb5b9233644ee3e1579518fe9: Status 404 returned error can't find the container with id 72966f1dbddeb6aa1d9643bd82e3aef651da682cb5b9233644ee3e1579518fe9
Oct 30 22:15:01.314650 ip-10-0-130-60 kubenswrapper[5061]: W1030 22:15:01.314628    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3787aeb2_4e6e_4855_8be8_137ebcd39a26.slice/crio-753bd4d64d3f6d0357ee517d614c3787d67381e5ba79679cc9f476c2c3c5ad6b WatchSource:0}: Error finding container 753bd4d64d3f6d0357ee517d614c3787d67381e5ba79679cc9f476c2c3c5ad6b: Status 404 returned error can't find the container with id 753bd4d64d3f6d0357ee517d614c3787d67381e5ba79679cc9f476c2c3c5ad6b
Oct 30 22:15:01.384378 ip-10-0-130-60 kubenswrapper[5061]: W1030 22:15:01.384351    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod777f3189_c018_4cdb_92ba_eeb10b71e868.slice/crio-69df25cba20b08dd834f32fd4448cf84312170dcad2ddee4dea97fed34ef3ad5 WatchSource:0}: Error finding container 69df25cba20b08dd834f32fd4448cf84312170dcad2ddee4dea97fed34ef3ad5: Status 404 returned error can't find the container with id 69df25cba20b08dd834f32fd4448cf84312170dcad2ddee4dea97fed34ef3ad5
Oct 30 22:15:01.384610 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:01.384508    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-25de17-29889975-qk5j5"]
Oct 30 22:15:01.448243 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:01.448202    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5fef81-29889975-fdwmx"]
Oct 30 22:15:01.613711 ip-10-0-130-60 kubenswrapper[5061]: W1030 22:15:01.613679    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbaff831_8881_4762_8332_25c8eb4c2a28.slice/crio-22fa96e22af060d2623e1588a6083c0a2f5f39c2643f92231453fb16fe142549 WatchSource:0}: Error finding container 22fa96e22af060d2623e1588a6083c0a2f5f39c2643f92231453fb16fe142549: Status 404 returned error can't find the container with id 22fa96e22af060d2623e1588a6083c0a2f5f39c2643f92231453fb16fe142549
Oct 30 22:15:01.836707 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:01.836672    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-411ff1-29889975-fc8l4" event={"ID":"52312d9d-e2f8-4ad3-bce1-53ffc5b08d4f","Type":"ContainerStarted","Data":"030ec511c379cfeb5123d2e3a2fee91af15a7c9d7bd6ac327f186c6114c8f84b"}
Oct 30 22:15:01.837959 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:01.837938    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-dq9xr" event={"ID":"f220141f-4150-4f9c-9e14-631777d0ac14","Type":"ContainerStarted","Data":"74ea822855504dd1f4fc36a274b5be614dcac462259ee2582463d739c242d447"}
Oct 30 22:15:01.839119 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:01.839098    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e66edb-29889975-flzf6" event={"ID":"8b4f58e4-2611-45f1-b431-a33ce392d964","Type":"ContainerStarted","Data":"010d1790a0b89cf36b7c5a2c6dd7d94d05ff7a79caf7401a4fd26c5fe49a8e64"}
Oct 30 22:15:01.840509 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:01.840491    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5402dd-29889975-jjcmb" event={"ID":"0ba41c1f-1fc3-4a32-967c-2d86878e2887","Type":"ContainerStarted","Data":"e581baed94e90e053ce34477ffc4a24cade12ff70246e04c1f3f2280f626aef2"}
Oct 30 22:15:01.840610 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:01.840536    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5402dd-29889975-jjcmb" event={"ID":"0ba41c1f-1fc3-4a32-967c-2d86878e2887","Type":"ContainerStarted","Data":"a4f7142dc6a02edfb7a68046d38c0650e316c7a877c46d032784c3f5810f1c61"}
Oct 30 22:15:01.842068 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:01.842045    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-6ea8da-29889975-r8jsf" event={"ID":"08aac4a2-7d84-4b74-bf47-dc849fd6e446","Type":"ContainerStarted","Data":"59d33243bf781c34bde318ab34cfbea262e138e45adca10c8de9543f130bfcf7"}
Oct 30 22:15:01.842162 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:01.842097    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-6ea8da-29889975-r8jsf" event={"ID":"08aac4a2-7d84-4b74-bf47-dc849fd6e446","Type":"ContainerStarted","Data":"ef79113642d3e216ccfa7fe792ee52b8a153464d312307fea798ce4c7d0d0888"}
Oct 30 22:15:01.843290 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:01.843268    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-25de17-29889975-qk5j5" event={"ID":"777f3189-c018-4cdb-92ba-eeb10b71e868","Type":"ContainerStarted","Data":"69df25cba20b08dd834f32fd4448cf84312170dcad2ddee4dea97fed34ef3ad5"}
Oct 30 22:15:01.844456 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:01.844435    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-4d8041-29889975-fn5gn" event={"ID":"42adc96b-1b21-484e-b6fa-51bbf1127ac0","Type":"ContainerStarted","Data":"75e1e5ca3c4f2da04268187cf485f66fce8f323743e6227061689bc2cefa18ed"}
Oct 30 22:15:01.845658 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:01.845636    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5fef81-29889975-fdwmx" event={"ID":"cbaff831-8881-4762-8332-25c8eb4c2a28","Type":"ContainerStarted","Data":"22fa96e22af060d2623e1588a6083c0a2f5f39c2643f92231453fb16fe142549"}
Oct 30 22:15:01.846942 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:01.846923    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-778d39d3-29889975-mct28" event={"ID":"3787aeb2-4e6e-4855-8be8-137ebcd39a26","Type":"ContainerStarted","Data":"753bd4d64d3f6d0357ee517d614c3787d67381e5ba79679cc9f476c2c3c5ad6b"}
Oct 30 22:15:01.847972 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:01.847953    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-02bd3b-29889975-jjfj2" event={"ID":"06e58c0b-aa5c-4f1e-8228-7070a03657bb","Type":"ContainerStarted","Data":"72966f1dbddeb6aa1d9643bd82e3aef651da682cb5b9233644ee3e1579518fe9"}
Oct 30 22:15:01.959767 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:01.959707    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5402dd-29889975-jjcmb" podStartSLOduration=1.9596902360000001 podStartE2EDuration="1.959690236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:01.955414918 +0000 UTC m=+1955.568184658" watchObservedRunningTime="2026-10-30 22:15:01.959690236 +0000 UTC m=+1955.572459995"
Oct 30 22:15:02.857206 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:02.857176    5061 generic.go:356] "Generic (PLEG): container finished" podID="f220141f-4150-4f9c-9e14-631777d0ac14" containerID="302fde95f633a74b616a92e6b952fc7bbb6f7b983c9f178d4b767363a8179b3d" exitCode=60
Oct 30 22:15:02.857592 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:02.857224    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-dq9xr" event={"ID":"f220141f-4150-4f9c-9e14-631777d0ac14","Type":"ContainerDied","Data":"302fde95f633a74b616a92e6b952fc7bbb6f7b983c9f178d4b767363a8179b3d"}
Oct 30 22:15:02.857643 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:02.857594    5061 scope.go:122] "RemoveContainer" containerID="302fde95f633a74b616a92e6b952fc7bbb6f7b983c9f178d4b767363a8179b3d"
Oct 30 22:15:02.859202 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:02.859177    5061 generic.go:356] "Generic (PLEG): container finished" podID="8b4f58e4-2611-45f1-b431-a33ce392d964" containerID="279d306111a36159c371bc9999bc5a043d3597931d99a1762b7ad1aa79c25564" exitCode=60
Oct 30 22:15:02.859310 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:02.859247    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e66edb-29889975-flzf6" event={"ID":"8b4f58e4-2611-45f1-b431-a33ce392d964","Type":"ContainerDied","Data":"279d306111a36159c371bc9999bc5a043d3597931d99a1762b7ad1aa79c25564"}
Oct 30 22:15:02.859576 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:02.859554    5061 scope.go:122] "RemoveContainer" containerID="279d306111a36159c371bc9999bc5a043d3597931d99a1762b7ad1aa79c25564"
Oct 30 22:15:02.861246 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:02.861200    5061 generic.go:356] "Generic (PLEG): container finished" podID="0ba41c1f-1fc3-4a32-967c-2d86878e2887" containerID="e581baed94e90e053ce34477ffc4a24cade12ff70246e04c1f3f2280f626aef2" exitCode=60
Oct 30 22:15:02.861297 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:02.861274    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5402dd-29889975-jjcmb" event={"ID":"0ba41c1f-1fc3-4a32-967c-2d86878e2887","Type":"ContainerDied","Data":"e581baed94e90e053ce34477ffc4a24cade12ff70246e04c1f3f2280f626aef2"}
Oct 30 22:15:02.861554 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:02.861500    5061 scope.go:122] "RemoveContainer" containerID="e581baed94e90e053ce34477ffc4a24cade12ff70246e04c1f3f2280f626aef2"
Oct 30 22:15:02.863086 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:02.863058    5061 generic.go:356] "Generic (PLEG): container finished" podID="08aac4a2-7d84-4b74-bf47-dc849fd6e446" containerID="59d33243bf781c34bde318ab34cfbea262e138e45adca10c8de9543f130bfcf7" exitCode=60
Oct 30 22:15:02.863176 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:02.863157    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-6ea8da-29889975-r8jsf" event={"ID":"08aac4a2-7d84-4b74-bf47-dc849fd6e446","Type":"ContainerDied","Data":"59d33243bf781c34bde318ab34cfbea262e138e45adca10c8de9543f130bfcf7"}
Oct 30 22:15:02.863311 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:02.863294    5061 scope.go:122] "RemoveContainer" containerID="59d33243bf781c34bde318ab34cfbea262e138e45adca10c8de9543f130bfcf7"
Oct 30 22:15:02.864813 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:02.864794    5061 generic.go:356] "Generic (PLEG): container finished" podID="777f3189-c018-4cdb-92ba-eeb10b71e868" containerID="cf78e4e7f2fc74ca3cfa9903ead7e84f7cd66929d38315a6528c024f212ceb33" exitCode=60
Oct 30 22:15:02.864920 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:02.864871    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-25de17-29889975-qk5j5" event={"ID":"777f3189-c018-4cdb-92ba-eeb10b71e868","Type":"ContainerDied","Data":"cf78e4e7f2fc74ca3cfa9903ead7e84f7cd66929d38315a6528c024f212ceb33"}
Oct 30 22:15:02.865035 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:02.865021    5061 scope.go:122] "RemoveContainer" containerID="cf78e4e7f2fc74ca3cfa9903ead7e84f7cd66929d38315a6528c024f212ceb33"
Oct 30 22:15:02.866414 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:02.866393    5061 generic.go:356] "Generic (PLEG): container finished" podID="42adc96b-1b21-484e-b6fa-51bbf1127ac0" containerID="76bcd8bc26c53fe26008399d498907010463d6f95382ef99d7c710f5b7314c1b" exitCode=60
Oct 30 22:15:02.866501 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:02.866476    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-4d8041-29889975-fn5gn" event={"ID":"42adc96b-1b21-484e-b6fa-51bbf1127ac0","Type":"ContainerDied","Data":"76bcd8bc26c53fe26008399d498907010463d6f95382ef99d7c710f5b7314c1b"}
Oct 30 22:15:02.866654 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:02.866638    5061 scope.go:122] "RemoveContainer" containerID="76bcd8bc26c53fe26008399d498907010463d6f95382ef99d7c710f5b7314c1b"
Oct 30 22:15:02.868153 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:02.868132    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5fef81-29889975-fdwmx" event={"ID":"cbaff831-8881-4762-8332-25c8eb4c2a28","Type":"ContainerStarted","Data":"8d5da1df557521087e19e64752f3b24d967743dcb9a049e1b0a065b393cb650a"}
Oct 30 22:15:02.870277 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:02.870255    5061 generic.go:356] "Generic (PLEG): container finished" podID="3787aeb2-4e6e-4855-8be8-137ebcd39a26" containerID="6e72f83fbff2b3407e689a58c7f34aa506873c8883e024154a0c925be7b826d9" exitCode=60
Oct 30 22:15:02.870386 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:02.870371    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-778d39d3-29889975-mct28" event={"ID":"3787aeb2-4e6e-4855-8be8-137ebcd39a26","Type":"ContainerDied","Data":"6e72f83fbff2b3407e689a58c7f34aa506873c8883e024154a0c925be7b826d9"}
Oct 30 22:15:02.870706 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:02.870684    5061 scope.go:122] "RemoveContainer" containerID="6e72f83fbff2b3407e689a58c7f34aa506873c8883e024154a0c925be7b826d9"
Oct 30 22:15:02.877913 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:02.877883    5061 generic.go:356] "Generic (PLEG): container finished" podID="06e58c0b-aa5c-4f1e-8228-7070a03657bb" containerID="c9b71e3822ab93e7322a30d5c3fbcff51a106acfad828794c2775de9794b502a" exitCode=60
Oct 30 22:15:02.877995 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:02.877912    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-02bd3b-29889975-jjfj2" event={"ID":"06e58c0b-aa5c-4f1e-8228-7070a03657bb","Type":"ContainerDied","Data":"c9b71e3822ab93e7322a30d5c3fbcff51a106acfad828794c2775de9794b502a"}
Oct 30 22:15:02.878181 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:02.878164    5061 scope.go:122] "RemoveContainer" containerID="c9b71e3822ab93e7322a30d5c3fbcff51a106acfad828794c2775de9794b502a"
Oct 30 22:15:02.880677 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:02.880653    5061 generic.go:356] "Generic (PLEG): container finished" podID="52312d9d-e2f8-4ad3-bce1-53ffc5b08d4f" containerID="988f1bb2b5461fbcb89192f1369e09871c38bbeccd3754c57a13f03094b8fd92" exitCode=60
Oct 30 22:15:02.880780 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:02.880703    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-411ff1-29889975-fc8l4" event={"ID":"52312d9d-e2f8-4ad3-bce1-53ffc5b08d4f","Type":"ContainerDied","Data":"988f1bb2b5461fbcb89192f1369e09871c38bbeccd3754c57a13f03094b8fd92"}
Oct 30 22:15:02.880967 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:02.880949    5061 scope.go:122] "RemoveContainer" containerID="988f1bb2b5461fbcb89192f1369e09871c38bbeccd3754c57a13f03094b8fd92"
Oct 30 22:15:02.971797 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:02.971707    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-6ea8da-29889975-r8jsf" podStartSLOduration=2.971687147 podStartE2EDuration="2.971687147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:02.070640301 +0000 UTC m=+1955.683410062" watchObservedRunningTime="2026-10-30 22:15:02.971687147 +0000 UTC m=+1956.584456906"
Oct 30 22:15:03.633347 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:03.633284    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5fef81-29889975-fdwmx" podStartSLOduration=3.633264837 podStartE2EDuration="3.633264837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:03.631585017 +0000 UTC m=+1957.244354793" watchObservedRunningTime="2026-10-30 22:15:03.633264837 +0000 UTC m=+1957.246034596"
Oct 30 22:15:03.890788 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:03.890752    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-dq9xr" event={"ID":"f220141f-4150-4f9c-9e14-631777d0ac14","Type":"ContainerStarted","Data":"087ef8ed22156e9fa524a056c90869a98bab01ec17606047bd67904711618db5"}
Oct 30 22:15:03.892673 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:03.892647    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e66edb-29889975-flzf6" event={"ID":"8b4f58e4-2611-45f1-b431-a33ce392d964","Type":"ContainerStarted","Data":"fb3dc4c5756703c6bf6bb73c15a55aee39110ba2c19f6c8536de6603631aaec7"}
Oct 30 22:15:03.894606 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:03.894584    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5402dd-29889975-jjcmb" event={"ID":"0ba41c1f-1fc3-4a32-967c-2d86878e2887","Type":"ContainerStarted","Data":"844a1473d20b228d7e2bae4c60e35dff3bc384aa51536a79cb8f3c8527fbbb75"}
Oct 30 22:15:03.896348 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:03.896328    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-6ea8da-29889975-r8jsf" event={"ID":"08aac4a2-7d84-4b74-bf47-dc849fd6e446","Type":"ContainerStarted","Data":"82dddbddcfcce85bc3d85b3bf22ef211a56c6bc33ad80549fee2923e0cdf341b"}
Oct 30 22:15:03.898459 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:03.898435    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-25de17-29889975-qk5j5" event={"ID":"777f3189-c018-4cdb-92ba-eeb10b71e868","Type":"ContainerStarted","Data":"04e15d99ade3218213acf362ff00652471c87c00ca5e1c6009acef8b49ea46c8"}
Oct 30 22:15:03.900512 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:03.900487    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-4d8041-29889975-fn5gn" event={"ID":"42adc96b-1b21-484e-b6fa-51bbf1127ac0","Type":"ContainerStarted","Data":"7459dbb94cf3d411470df36dcba68ec8b82760981fd11d816dbfe6163c18b5da"}
Oct 30 22:15:03.902032 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:03.902010    5061 generic.go:356] "Generic (PLEG): container finished" podID="cbaff831-8881-4762-8332-25c8eb4c2a28" containerID="8d5da1df557521087e19e64752f3b24d967743dcb9a049e1b0a065b393cb650a" exitCode=60
Oct 30 22:15:03.902123 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:03.902075    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5fef81-29889975-fdwmx" event={"ID":"cbaff831-8881-4762-8332-25c8eb4c2a28","Type":"ContainerDied","Data":"8d5da1df557521087e19e64752f3b24d967743dcb9a049e1b0a065b393cb650a"}
Oct 30 22:15:03.902291 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:03.902269    5061 scope.go:122] "RemoveContainer" containerID="8d5da1df557521087e19e64752f3b24d967743dcb9a049e1b0a065b393cb650a"
Oct 30 22:15:03.904480 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:03.904299    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-778d39d3-29889975-mct28" event={"ID":"3787aeb2-4e6e-4855-8be8-137ebcd39a26","Type":"ContainerStarted","Data":"847811d9d3d28d49a7bae39801791c1de8cf92f8506dca3ea4bfe8cb2fa90c1f"}
Oct 30 22:15:03.906632 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:03.906606    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-02bd3b-29889975-jjfj2" event={"ID":"06e58c0b-aa5c-4f1e-8228-7070a03657bb","Type":"ContainerStarted","Data":"09a0f8e9190d2cfa49446ad8a6c2c9b0e5f26e73843fb649cf6c592faeec3f05"}
Oct 30 22:15:03.912413 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:03.912387    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-411ff1-29889975-fc8l4" event={"ID":"52312d9d-e2f8-4ad3-bce1-53ffc5b08d4f","Type":"ContainerStarted","Data":"4e661358350ceeaed21dd29b7429217514d11f83cb0e55a4664963b8fcf1a144"}
Oct 30 22:15:04.222582 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:04.222494    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e66edb-29889975-flzf6" podStartSLOduration=4.22246454 podStartE2EDuration="4.22246454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:04.210044058 +0000 UTC m=+1957.822813830" watchObservedRunningTime="2026-10-30 22:15:04.22246454 +0000 UTC m=+1957.835234302"
Oct 30 22:15:04.323337 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:04.323289    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-02bd3b-29889975-jjfj2" podStartSLOduration=4.323275768 podStartE2EDuration="4.323275768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:04.32031405 +0000 UTC m=+1957.933083837" watchObservedRunningTime="2026-10-30 22:15:04.323275768 +0000 UTC m=+1957.936045525"
Oct 30 22:15:04.432138 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:04.432090    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-25de17-29889975-qk5j5" podStartSLOduration=4.432075382 podStartE2EDuration="4.432075382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:04.429293702 +0000 UTC m=+1958.042063460" watchObservedRunningTime="2026-10-30 22:15:04.432075382 +0000 UTC m=+1958.044845140"
Oct 30 22:15:04.756379 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:04.756329    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-411ff1-29889975-fc8l4" podStartSLOduration=4.756311857 podStartE2EDuration="4.756311857s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:04.752886495 +0000 UTC m=+1958.365656276" watchObservedRunningTime="2026-10-30 22:15:04.756311857 +0000 UTC m=+1958.369081612"
Oct 30 22:15:04.865132 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:04.865076    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-4d8041-29889975-fn5gn" podStartSLOduration=4.8650630790000005 podStartE2EDuration="4.865063079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:04.862777881 +0000 UTC m=+1958.475547627" watchObservedRunningTime="2026-10-30 22:15:04.865063079 +0000 UTC m=+1958.477832852"
Oct 30 22:15:04.923868 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:04.923788    5061 generic.go:356] "Generic (PLEG): container finished" podID="52312d9d-e2f8-4ad3-bce1-53ffc5b08d4f" containerID="4e661358350ceeaed21dd29b7429217514d11f83cb0e55a4664963b8fcf1a144" exitCode=60
Oct 30 22:15:04.924276 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:04.923873    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-411ff1-29889975-fc8l4" event={"ID":"52312d9d-e2f8-4ad3-bce1-53ffc5b08d4f","Type":"ContainerDied","Data":"4e661358350ceeaed21dd29b7429217514d11f83cb0e55a4664963b8fcf1a144"}
Oct 30 22:15:04.924276 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:04.923926    5061 scope.go:122] "RemoveContainer" containerID="988f1bb2b5461fbcb89192f1369e09871c38bbeccd3754c57a13f03094b8fd92"
Oct 30 22:15:04.924276 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:04.924189    5061 scope.go:122] "RemoveContainer" containerID="4e661358350ceeaed21dd29b7429217514d11f83cb0e55a4664963b8fcf1a144"
Oct 30 22:15:04.924640 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:15:04.924607    5061 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-411ff1-29889975-fc8l4_odh-ai-gateway-infra(52312d9d-e2f8-4ad3-bce1-53ffc5b08d4f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-411ff1-29889975-fc8l4" podUID="52312d9d-e2f8-4ad3-bce1-53ffc5b08d4f"
Oct 30 22:15:04.925889 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:04.925864    5061 generic.go:356] "Generic (PLEG): container finished" podID="f220141f-4150-4f9c-9e14-631777d0ac14" containerID="087ef8ed22156e9fa524a056c90869a98bab01ec17606047bd67904711618db5" exitCode=60
Oct 30 22:15:04.926000 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:04.925951    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-dq9xr" event={"ID":"f220141f-4150-4f9c-9e14-631777d0ac14","Type":"ContainerDied","Data":"087ef8ed22156e9fa524a056c90869a98bab01ec17606047bd67904711618db5"}
Oct 30 22:15:04.926214 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:04.926196    5061 scope.go:122] "RemoveContainer" containerID="087ef8ed22156e9fa524a056c90869a98bab01ec17606047bd67904711618db5"
Oct 30 22:15:04.926563 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:15:04.926513    5061 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889975-dq9xr_odh-ai-gateway-infra(f220141f-4150-4f9c-9e14-631777d0ac14)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-dq9xr" podUID="f220141f-4150-4f9c-9e14-631777d0ac14"
Oct 30 22:15:04.927915 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:04.927894    5061 generic.go:356] "Generic (PLEG): container finished" podID="8b4f58e4-2611-45f1-b431-a33ce392d964" containerID="fb3dc4c5756703c6bf6bb73c15a55aee39110ba2c19f6c8536de6603631aaec7" exitCode=60
Oct 30 22:15:04.928010 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:04.927977    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e66edb-29889975-flzf6" event={"ID":"8b4f58e4-2611-45f1-b431-a33ce392d964","Type":"ContainerDied","Data":"fb3dc4c5756703c6bf6bb73c15a55aee39110ba2c19f6c8536de6603631aaec7"}
Oct 30 22:15:04.928167 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:04.928150    5061 scope.go:122] "RemoveContainer" containerID="fb3dc4c5756703c6bf6bb73c15a55aee39110ba2c19f6c8536de6603631aaec7"
Oct 30 22:15:04.928457 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:15:04.928433    5061 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-e66edb-29889975-flzf6_odh-ai-gateway-infra(8b4f58e4-2611-45f1-b431-a33ce392d964)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e66edb-29889975-flzf6" podUID="8b4f58e4-2611-45f1-b431-a33ce392d964"
Oct 30 22:15:04.929932 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:04.929913    5061 generic.go:356] "Generic (PLEG): container finished" podID="0ba41c1f-1fc3-4a32-967c-2d86878e2887" containerID="844a1473d20b228d7e2bae4c60e35dff3bc384aa51536a79cb8f3c8527fbbb75" exitCode=60
Oct 30 22:15:04.930021 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:04.929976    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5402dd-29889975-jjcmb" event={"ID":"0ba41c1f-1fc3-4a32-967c-2d86878e2887","Type":"ContainerDied","Data":"844a1473d20b228d7e2bae4c60e35dff3bc384aa51536a79cb8f3c8527fbbb75"}
Oct 30 22:15:04.930153 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:04.930139    5061 scope.go:122] "RemoveContainer" containerID="844a1473d20b228d7e2bae4c60e35dff3bc384aa51536a79cb8f3c8527fbbb75"
Oct 30 22:15:04.930428 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:15:04.930405    5061 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-5402dd-29889975-jjcmb_odh-ai-gateway-infra(0ba41c1f-1fc3-4a32-967c-2d86878e2887)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5402dd-29889975-jjcmb" podUID="0ba41c1f-1fc3-4a32-967c-2d86878e2887"
Oct 30 22:15:04.931978 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:04.931957    5061 generic.go:356] "Generic (PLEG): container finished" podID="08aac4a2-7d84-4b74-bf47-dc849fd6e446" containerID="82dddbddcfcce85bc3d85b3bf22ef211a56c6bc33ad80549fee2923e0cdf341b" exitCode=60
Oct 30 22:15:04.932045 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:04.932030    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-6ea8da-29889975-r8jsf" event={"ID":"08aac4a2-7d84-4b74-bf47-dc849fd6e446","Type":"ContainerDied","Data":"82dddbddcfcce85bc3d85b3bf22ef211a56c6bc33ad80549fee2923e0cdf341b"}
Oct 30 22:15:04.932204 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:04.932177    5061 scope.go:122] "RemoveContainer" containerID="82dddbddcfcce85bc3d85b3bf22ef211a56c6bc33ad80549fee2923e0cdf341b"
Oct 30 22:15:04.932485 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:15:04.932463    5061 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-6ea8da-29889975-r8jsf_odh-ai-gateway-infra(08aac4a2-7d84-4b74-bf47-dc849fd6e446)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-6ea8da-29889975-r8jsf" podUID="08aac4a2-7d84-4b74-bf47-dc849fd6e446"
Oct 30 22:15:04.934081 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:04.934063    5061 generic.go:356] "Generic (PLEG): container finished" podID="777f3189-c018-4cdb-92ba-eeb10b71e868" containerID="04e15d99ade3218213acf362ff00652471c87c00ca5e1c6009acef8b49ea46c8" exitCode=60
Oct 30 22:15:04.934174 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:04.934151    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-25de17-29889975-qk5j5" event={"ID":"777f3189-c018-4cdb-92ba-eeb10b71e868","Type":"ContainerDied","Data":"04e15d99ade3218213acf362ff00652471c87c00ca5e1c6009acef8b49ea46c8"}
Oct 30 22:15:04.934349 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:04.934329    5061 scope.go:122] "RemoveContainer" containerID="04e15d99ade3218213acf362ff00652471c87c00ca5e1c6009acef8b49ea46c8"
Oct 30 22:15:04.934609 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:15:04.934589    5061 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-25de17-29889975-qk5j5_odh-ai-gateway-infra(777f3189-c018-4cdb-92ba-eeb10b71e868)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-25de17-29889975-qk5j5" podUID="777f3189-c018-4cdb-92ba-eeb10b71e868"
Oct 30 22:15:04.936041 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:04.936022    5061 generic.go:356] "Generic (PLEG): container finished" podID="42adc96b-1b21-484e-b6fa-51bbf1127ac0" containerID="7459dbb94cf3d411470df36dcba68ec8b82760981fd11d816dbfe6163c18b5da" exitCode=60
Oct 30 22:15:04.936105 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:04.936090    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-4d8041-29889975-fn5gn" event={"ID":"42adc96b-1b21-484e-b6fa-51bbf1127ac0","Type":"ContainerDied","Data":"7459dbb94cf3d411470df36dcba68ec8b82760981fd11d816dbfe6163c18b5da"}
Oct 30 22:15:04.936289 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:04.936271    5061 scope.go:122] "RemoveContainer" containerID="7459dbb94cf3d411470df36dcba68ec8b82760981fd11d816dbfe6163c18b5da"
Oct 30 22:15:04.936567 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:15:04.936513    5061 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-4d8041-29889975-fn5gn_odh-ai-gateway-infra(42adc96b-1b21-484e-b6fa-51bbf1127ac0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-4d8041-29889975-fn5gn" podUID="42adc96b-1b21-484e-b6fa-51bbf1127ac0"
Oct 30 22:15:04.938072 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:04.938051    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5fef81-29889975-fdwmx" event={"ID":"cbaff831-8881-4762-8332-25c8eb4c2a28","Type":"ContainerStarted","Data":"31464249c4951adb2f8ba7b6f6eec05cc32771225c8cc082b82dee5caeec4f25"}
Oct 30 22:15:04.939521 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:04.939505    5061 scope.go:122] "RemoveContainer" containerID="302fde95f633a74b616a92e6b952fc7bbb6f7b983c9f178d4b767363a8179b3d"
Oct 30 22:15:04.939936 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:04.939919    5061 generic.go:356] "Generic (PLEG): container finished" podID="3787aeb2-4e6e-4855-8be8-137ebcd39a26" containerID="847811d9d3d28d49a7bae39801791c1de8cf92f8506dca3ea4bfe8cb2fa90c1f" exitCode=60
Oct 30 22:15:04.940042 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:04.940018    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-778d39d3-29889975-mct28" event={"ID":"3787aeb2-4e6e-4855-8be8-137ebcd39a26","Type":"ContainerDied","Data":"847811d9d3d28d49a7bae39801791c1de8cf92f8506dca3ea4bfe8cb2fa90c1f"}
Oct 30 22:15:04.940219 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:04.940200    5061 scope.go:122] "RemoveContainer" containerID="847811d9d3d28d49a7bae39801791c1de8cf92f8506dca3ea4bfe8cb2fa90c1f"
Oct 30 22:15:04.940439 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:15:04.940413    5061 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-778d39d3-29889975-mct28_odh-ai-gateway-infra(3787aeb2-4e6e-4855-8be8-137ebcd39a26)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-778d39d3-29889975-mct28" podUID="3787aeb2-4e6e-4855-8be8-137ebcd39a26"
Oct 30 22:15:04.941922 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:04.941898    5061 generic.go:356] "Generic (PLEG): container finished" podID="06e58c0b-aa5c-4f1e-8228-7070a03657bb" containerID="09a0f8e9190d2cfa49446ad8a6c2c9b0e5f26e73843fb649cf6c592faeec3f05" exitCode=60
Oct 30 22:15:04.942033 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:04.941940    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-02bd3b-29889975-jjfj2" event={"ID":"06e58c0b-aa5c-4f1e-8228-7070a03657bb","Type":"ContainerDied","Data":"09a0f8e9190d2cfa49446ad8a6c2c9b0e5f26e73843fb649cf6c592faeec3f05"}
Oct 30 22:15:04.942267 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:04.942250    5061 scope.go:122] "RemoveContainer" containerID="09a0f8e9190d2cfa49446ad8a6c2c9b0e5f26e73843fb649cf6c592faeec3f05"
Oct 30 22:15:04.942490 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:15:04.942473    5061 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-02bd3b-29889975-jjfj2_odh-ai-gateway-infra(06e58c0b-aa5c-4f1e-8228-7070a03657bb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-02bd3b-29889975-jjfj2" podUID="06e58c0b-aa5c-4f1e-8228-7070a03657bb"
Oct 30 22:15:04.954129 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:04.954106    5061 scope.go:122] "RemoveContainer" containerID="279d306111a36159c371bc9999bc5a043d3597931d99a1762b7ad1aa79c25564"
Oct 30 22:15:04.976413 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:04.976356    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-778d39d3-29889975-mct28" podStartSLOduration=4.976337419 podStartE2EDuration="4.976337419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:04.973223919 +0000 UTC m=+1958.585993701" watchObservedRunningTime="2026-10-30 22:15:04.976337419 +0000 UTC m=+1958.589107179"
Oct 30 22:15:04.981282 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:04.981261    5061 scope.go:122] "RemoveContainer" containerID="e581baed94e90e053ce34477ffc4a24cade12ff70246e04c1f3f2280f626aef2"
Oct 30 22:15:04.996081 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:04.996063    5061 scope.go:122] "RemoveContainer" containerID="59d33243bf781c34bde318ab34cfbea262e138e45adca10c8de9543f130bfcf7"
Oct 30 22:15:05.017119 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:05.017091    5061 scope.go:122] "RemoveContainer" containerID="cf78e4e7f2fc74ca3cfa9903ead7e84f7cd66929d38315a6528c024f212ceb33"
Oct 30 22:15:05.030512 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:05.030484    5061 scope.go:122] "RemoveContainer" containerID="76bcd8bc26c53fe26008399d498907010463d6f95382ef99d7c710f5b7314c1b"
Oct 30 22:15:05.044786 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:05.044764    5061 scope.go:122] "RemoveContainer" containerID="6e72f83fbff2b3407e689a58c7f34aa506873c8883e024154a0c925be7b826d9"
Oct 30 22:15:05.058596 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:05.058574    5061 scope.go:122] "RemoveContainer" containerID="c9b71e3822ab93e7322a30d5c3fbcff51a106acfad828794c2775de9794b502a"
Oct 30 22:15:05.083784 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:05.083730    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-dq9xr" podStartSLOduration=5.083717648 podStartE2EDuration="5.083717648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:05.080785914 +0000 UTC m=+1958.693555726" watchObservedRunningTime="2026-10-30 22:15:05.083717648 +0000 UTC m=+1958.696487406"
Oct 30 22:15:05.951766 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:05.951734    5061 scope.go:122] "RemoveContainer" containerID="4e661358350ceeaed21dd29b7429217514d11f83cb0e55a4664963b8fcf1a144"
Oct 30 22:15:05.952317 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:15:05.952282    5061 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-411ff1-29889975-fc8l4_odh-ai-gateway-infra(52312d9d-e2f8-4ad3-bce1-53ffc5b08d4f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-411ff1-29889975-fc8l4" podUID="52312d9d-e2f8-4ad3-bce1-53ffc5b08d4f"
Oct 30 22:15:05.953921 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:05.953890    5061 scope.go:122] "RemoveContainer" containerID="087ef8ed22156e9fa524a056c90869a98bab01ec17606047bd67904711618db5"
Oct 30 22:15:05.954283 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:15:05.954257    5061 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889975-dq9xr_odh-ai-gateway-infra(f220141f-4150-4f9c-9e14-631777d0ac14)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-dq9xr" podUID="f220141f-4150-4f9c-9e14-631777d0ac14"
Oct 30 22:15:05.970768 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:05.956197    5061 scope.go:122] "RemoveContainer" containerID="fb3dc4c5756703c6bf6bb73c15a55aee39110ba2c19f6c8536de6603631aaec7"
Oct 30 22:15:05.970768 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:15:05.956555    5061 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-e66edb-29889975-flzf6_odh-ai-gateway-infra(8b4f58e4-2611-45f1-b431-a33ce392d964)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e66edb-29889975-flzf6" podUID="8b4f58e4-2611-45f1-b431-a33ce392d964"
Oct 30 22:15:05.970768 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:05.958601    5061 scope.go:122] "RemoveContainer" containerID="844a1473d20b228d7e2bae4c60e35dff3bc384aa51536a79cb8f3c8527fbbb75"
Oct 30 22:15:05.970768 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:15:05.958859    5061 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-5402dd-29889975-jjcmb_odh-ai-gateway-infra(0ba41c1f-1fc3-4a32-967c-2d86878e2887)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5402dd-29889975-jjcmb" podUID="0ba41c1f-1fc3-4a32-967c-2d86878e2887"
Oct 30 22:15:05.970768 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:05.961477    5061 scope.go:122] "RemoveContainer" containerID="82dddbddcfcce85bc3d85b3bf22ef211a56c6bc33ad80549fee2923e0cdf341b"
Oct 30 22:15:05.970768 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:15:05.961760    5061 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-6ea8da-29889975-r8jsf_odh-ai-gateway-infra(08aac4a2-7d84-4b74-bf47-dc849fd6e446)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-6ea8da-29889975-r8jsf" podUID="08aac4a2-7d84-4b74-bf47-dc849fd6e446"
Oct 30 22:15:05.970768 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:05.963422    5061 scope.go:122] "RemoveContainer" containerID="04e15d99ade3218213acf362ff00652471c87c00ca5e1c6009acef8b49ea46c8"
Oct 30 22:15:05.970768 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:15:05.963704    5061 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-25de17-29889975-qk5j5_odh-ai-gateway-infra(777f3189-c018-4cdb-92ba-eeb10b71e868)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-25de17-29889975-qk5j5" podUID="777f3189-c018-4cdb-92ba-eeb10b71e868"
Oct 30 22:15:05.970768 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:05.965405    5061 scope.go:122] "RemoveContainer" containerID="7459dbb94cf3d411470df36dcba68ec8b82760981fd11d816dbfe6163c18b5da"
Oct 30 22:15:05.970768 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:15:05.965684    5061 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-4d8041-29889975-fn5gn_odh-ai-gateway-infra(42adc96b-1b21-484e-b6fa-51bbf1127ac0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-4d8041-29889975-fn5gn" podUID="42adc96b-1b21-484e-b6fa-51bbf1127ac0"
Oct 30 22:15:05.970768 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:05.969132    5061 generic.go:356] "Generic (PLEG): container finished" podID="cbaff831-8881-4762-8332-25c8eb4c2a28" containerID="31464249c4951adb2f8ba7b6f6eec05cc32771225c8cc082b82dee5caeec4f25" exitCode=60
Oct 30 22:15:05.970768 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:05.969217    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5fef81-29889975-fdwmx" event={"ID":"cbaff831-8881-4762-8332-25c8eb4c2a28","Type":"ContainerDied","Data":"31464249c4951adb2f8ba7b6f6eec05cc32771225c8cc082b82dee5caeec4f25"}
Oct 30 22:15:05.970768 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:05.969275    5061 scope.go:122] "RemoveContainer" containerID="8d5da1df557521087e19e64752f3b24d967743dcb9a049e1b0a065b393cb650a"
Oct 30 22:15:05.970768 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:05.969454    5061 scope.go:122] "RemoveContainer" containerID="31464249c4951adb2f8ba7b6f6eec05cc32771225c8cc082b82dee5caeec4f25"
Oct 30 22:15:05.970768 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:15:05.969769    5061 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-5fef81-29889975-fdwmx_odh-ai-gateway-infra(cbaff831-8881-4762-8332-25c8eb4c2a28)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5fef81-29889975-fdwmx" podUID="cbaff831-8881-4762-8332-25c8eb4c2a28"
Oct 30 22:15:05.971979 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:05.971499    5061 scope.go:122] "RemoveContainer" containerID="847811d9d3d28d49a7bae39801791c1de8cf92f8506dca3ea4bfe8cb2fa90c1f"
Oct 30 22:15:05.971979 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:15:05.971780    5061 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-778d39d3-29889975-mct28_odh-ai-gateway-infra(3787aeb2-4e6e-4855-8be8-137ebcd39a26)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-778d39d3-29889975-mct28" podUID="3787aeb2-4e6e-4855-8be8-137ebcd39a26"
Oct 30 22:15:05.974163 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:05.974144    5061 scope.go:122] "RemoveContainer" containerID="09a0f8e9190d2cfa49446ad8a6c2c9b0e5f26e73843fb649cf6c592faeec3f05"
Oct 30 22:15:05.974422 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:15:05.974401    5061 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-02bd3b-29889975-jjfj2_odh-ai-gateway-infra(06e58c0b-aa5c-4f1e-8228-7070a03657bb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-02bd3b-29889975-jjfj2" podUID="06e58c0b-aa5c-4f1e-8228-7070a03657bb"
Oct 30 22:15:06.983930 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:06.983900    5061 scope.go:122] "RemoveContainer" containerID="31464249c4951adb2f8ba7b6f6eec05cc32771225c8cc082b82dee5caeec4f25"
Oct 30 22:15:06.984348 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:15:06.984300    5061 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-5fef81-29889975-fdwmx_odh-ai-gateway-infra(cbaff831-8881-4762-8332-25c8eb4c2a28)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5fef81-29889975-fdwmx" podUID="cbaff831-8881-4762-8332-25c8eb4c2a28"
Oct 30 22:15:13.308094 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:13.308059    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-n4bnf"]
Oct 30 22:15:13.308513 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:13.308327    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-n4bnf" podUID="799953d1-1dc8-421e-9e98-d2696d399ac2" containerName="istio-proxy" containerID="cri-o://8eddbddec2bf4fa136c7343e98b86ac5eb26c4de459abf34ac50a87be0e862a6" gracePeriod=30
Oct 30 22:15:13.834493 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:13.834459    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-5z645"]
Oct 30 22:15:13.834793 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:13.834763    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-5z645" podUID="a84e4890-ba61-49a5-b9b6-be9cf741d2d8" containerName="istio-proxy" containerID="cri-o://b4886f646e7e94afb16aff7aa126959dd64ed13c753ca92a3013d38214007232" gracePeriod=30
Oct 30 22:15:15.521819 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:15.521788    5061 scope.go:122] "RemoveContainer" containerID="087ef8ed22156e9fa524a056c90869a98bab01ec17606047bd67904711618db5"
Oct 30 22:15:15.522275 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:15.521987    5061 scope.go:122] "RemoveContainer" containerID="fb3dc4c5756703c6bf6bb73c15a55aee39110ba2c19f6c8536de6603631aaec7"
Oct 30 22:15:16.099615 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:16.099587    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-dq9xr" event={"ID":"f220141f-4150-4f9c-9e14-631777d0ac14","Type":"ContainerStarted","Data":"07fdf70a88a04ac7efc92c5e49d0ebd555edfb6f1566d39343b15b8293059b8c"}
Oct 30 22:15:16.101645 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:16.101622    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e66edb-29889975-flzf6" event={"ID":"8b4f58e4-2611-45f1-b431-a33ce392d964","Type":"ContainerStarted","Data":"4b9f57dc4c1733ce47e979b2d62225e7a7913ac075487123a961cfd2366c525f"}
Oct 30 22:15:16.263223 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:16.263154    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-0ce563-openshift-default-f75cc7fc9-445r7"]
Oct 30 22:15:16.267881 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:16.267861    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-0ce563-openshift-default-f75cc7fc9-445r7"
Oct 30 22:15:16.318166 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:16.318139    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-0ce563-openshift-default-dockercfg-jjwxv\""
Oct 30 22:15:16.339750 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:16.339719    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-0ce563-openshift-default-f75cc7fc9-445r7"]
Oct 30 22:15:16.361928 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:16.361899    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a5f5114-bc2f-48c9-af68-f2d981ec9ce5-workload-socket\") pod \"e2e-api-b-0ce563-openshift-default-f75cc7fc9-445r7\" (UID: \"0a5f5114-bc2f-48c9-af68-f2d981ec9ce5\") " pod="openshift-ingress/e2e-api-b-0ce563-openshift-default-f75cc7fc9-445r7"
Oct 30 22:15:16.362069 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:16.361966    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a5f5114-bc2f-48c9-af68-f2d981ec9ce5-istio-envoy\") pod \"e2e-api-b-0ce563-openshift-default-f75cc7fc9-445r7\" (UID: \"0a5f5114-bc2f-48c9-af68-f2d981ec9ce5\") " pod="openshift-ingress/e2e-api-b-0ce563-openshift-default-f75cc7fc9-445r7"
Oct 30 22:15:16.362069 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:16.362055    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a5f5114-bc2f-48c9-af68-f2d981ec9ce5-istio-podinfo\") pod \"e2e-api-b-0ce563-openshift-default-f75cc7fc9-445r7\" (UID: \"0a5f5114-bc2f-48c9-af68-f2d981ec9ce5\") " pod="openshift-ingress/e2e-api-b-0ce563-openshift-default-f75cc7fc9-445r7"
Oct 30 22:15:16.362185 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:16.362113    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a5f5114-bc2f-48c9-af68-f2d981ec9ce5-istio-data\") pod \"e2e-api-b-0ce563-openshift-default-f75cc7fc9-445r7\" (UID: \"0a5f5114-bc2f-48c9-af68-f2d981ec9ce5\") " pod="openshift-ingress/e2e-api-b-0ce563-openshift-default-f75cc7fc9-445r7"
Oct 30 22:15:16.362262 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:16.362229    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a5f5114-bc2f-48c9-af68-f2d981ec9ce5-istio-token\") pod \"e2e-api-b-0ce563-openshift-default-f75cc7fc9-445r7\" (UID: \"0a5f5114-bc2f-48c9-af68-f2d981ec9ce5\") " pod="openshift-ingress/e2e-api-b-0ce563-openshift-default-f75cc7fc9-445r7"
Oct 30 22:15:16.362393 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:16.362373    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a5f5114-bc2f-48c9-af68-f2d981ec9ce5-workload-certs\") pod \"e2e-api-b-0ce563-openshift-default-f75cc7fc9-445r7\" (UID: \"0a5f5114-bc2f-48c9-af68-f2d981ec9ce5\") " pod="openshift-ingress/e2e-api-b-0ce563-openshift-default-f75cc7fc9-445r7"
Oct 30 22:15:16.362469 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:16.362406    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5swlb\" (UniqueName: \"kubernetes.io/projected/0a5f5114-bc2f-48c9-af68-f2d981ec9ce5-kube-api-access-5swlb\") pod \"e2e-api-b-0ce563-openshift-default-f75cc7fc9-445r7\" (UID: \"0a5f5114-bc2f-48c9-af68-f2d981ec9ce5\") " pod="openshift-ingress/e2e-api-b-0ce563-openshift-default-f75cc7fc9-445r7"
Oct 30 22:15:16.362469 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:16.362443    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a5f5114-bc2f-48c9-af68-f2d981ec9ce5-credential-socket\") pod \"e2e-api-b-0ce563-openshift-default-f75cc7fc9-445r7\" (UID: \"0a5f5114-bc2f-48c9-af68-f2d981ec9ce5\") " pod="openshift-ingress/e2e-api-b-0ce563-openshift-default-f75cc7fc9-445r7"
Oct 30 22:15:16.362688 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:16.362663    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a5f5114-bc2f-48c9-af68-f2d981ec9ce5-istiod-ca-cert\") pod \"e2e-api-b-0ce563-openshift-default-f75cc7fc9-445r7\" (UID: \"0a5f5114-bc2f-48c9-af68-f2d981ec9ce5\") " pod="openshift-ingress/e2e-api-b-0ce563-openshift-default-f75cc7fc9-445r7"
Oct 30 22:15:16.463620 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:16.463584    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a5f5114-bc2f-48c9-af68-f2d981ec9ce5-istio-data\") pod \"e2e-api-b-0ce563-openshift-default-f75cc7fc9-445r7\" (UID: \"0a5f5114-bc2f-48c9-af68-f2d981ec9ce5\") " pod="openshift-ingress/e2e-api-b-0ce563-openshift-default-f75cc7fc9-445r7"
Oct 30 22:15:16.463777 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:16.463650    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a5f5114-bc2f-48c9-af68-f2d981ec9ce5-istio-token\") pod \"e2e-api-b-0ce563-openshift-default-f75cc7fc9-445r7\" (UID: \"0a5f5114-bc2f-48c9-af68-f2d981ec9ce5\") " pod="openshift-ingress/e2e-api-b-0ce563-openshift-default-f75cc7fc9-445r7"
Oct 30 22:15:16.463777 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:16.463748    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a5f5114-bc2f-48c9-af68-f2d981ec9ce5-workload-certs\") pod \"e2e-api-b-0ce563-openshift-default-f75cc7fc9-445r7\" (UID: \"0a5f5114-bc2f-48c9-af68-f2d981ec9ce5\") " pod="openshift-ingress/e2e-api-b-0ce563-openshift-default-f75cc7fc9-445r7"
Oct 30 22:15:16.463945 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:16.463778    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5swlb\" (UniqueName: \"kubernetes.io/projected/0a5f5114-bc2f-48c9-af68-f2d981ec9ce5-kube-api-access-5swlb\") pod \"e2e-api-b-0ce563-openshift-default-f75cc7fc9-445r7\" (UID: \"0a5f5114-bc2f-48c9-af68-f2d981ec9ce5\") " pod="openshift-ingress/e2e-api-b-0ce563-openshift-default-f75cc7fc9-445r7"
Oct 30 22:15:16.463945 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:16.463821    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a5f5114-bc2f-48c9-af68-f2d981ec9ce5-credential-socket\") pod \"e2e-api-b-0ce563-openshift-default-f75cc7fc9-445r7\" (UID: \"0a5f5114-bc2f-48c9-af68-f2d981ec9ce5\") " pod="openshift-ingress/e2e-api-b-0ce563-openshift-default-f75cc7fc9-445r7"
Oct 30 22:15:16.464066 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:16.463974    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a5f5114-bc2f-48c9-af68-f2d981ec9ce5-istiod-ca-cert\") pod \"e2e-api-b-0ce563-openshift-default-f75cc7fc9-445r7\" (UID: \"0a5f5114-bc2f-48c9-af68-f2d981ec9ce5\") " pod="openshift-ingress/e2e-api-b-0ce563-openshift-default-f75cc7fc9-445r7"
Oct 30 22:15:16.464066 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:16.463981    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a5f5114-bc2f-48c9-af68-f2d981ec9ce5-istio-data\") pod \"e2e-api-b-0ce563-openshift-default-f75cc7fc9-445r7\" (UID: \"0a5f5114-bc2f-48c9-af68-f2d981ec9ce5\") " pod="openshift-ingress/e2e-api-b-0ce563-openshift-default-f75cc7fc9-445r7"
Oct 30 22:15:16.464066 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:16.464010    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a5f5114-bc2f-48c9-af68-f2d981ec9ce5-workload-socket\") pod \"e2e-api-b-0ce563-openshift-default-f75cc7fc9-445r7\" (UID: \"0a5f5114-bc2f-48c9-af68-f2d981ec9ce5\") " pod="openshift-ingress/e2e-api-b-0ce563-openshift-default-f75cc7fc9-445r7"
Oct 30 22:15:16.464227 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:16.464066    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a5f5114-bc2f-48c9-af68-f2d981ec9ce5-workload-certs\") pod \"e2e-api-b-0ce563-openshift-default-f75cc7fc9-445r7\" (UID: \"0a5f5114-bc2f-48c9-af68-f2d981ec9ce5\") " pod="openshift-ingress/e2e-api-b-0ce563-openshift-default-f75cc7fc9-445r7"
Oct 30 22:15:16.464227 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:16.464088    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a5f5114-bc2f-48c9-af68-f2d981ec9ce5-istio-envoy\") pod \"e2e-api-b-0ce563-openshift-default-f75cc7fc9-445r7\" (UID: \"0a5f5114-bc2f-48c9-af68-f2d981ec9ce5\") " pod="openshift-ingress/e2e-api-b-0ce563-openshift-default-f75cc7fc9-445r7"
Oct 30 22:15:16.464227 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:16.464145    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a5f5114-bc2f-48c9-af68-f2d981ec9ce5-istio-podinfo\") pod \"e2e-api-b-0ce563-openshift-default-f75cc7fc9-445r7\" (UID: \"0a5f5114-bc2f-48c9-af68-f2d981ec9ce5\") " pod="openshift-ingress/e2e-api-b-0ce563-openshift-default-f75cc7fc9-445r7"
Oct 30 22:15:16.464360 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:16.464324    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a5f5114-bc2f-48c9-af68-f2d981ec9ce5-workload-socket\") pod \"e2e-api-b-0ce563-openshift-default-f75cc7fc9-445r7\" (UID: \"0a5f5114-bc2f-48c9-af68-f2d981ec9ce5\") " pod="openshift-ingress/e2e-api-b-0ce563-openshift-default-f75cc7fc9-445r7"
Oct 30 22:15:16.464360 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:16.464348    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a5f5114-bc2f-48c9-af68-f2d981ec9ce5-credential-socket\") pod \"e2e-api-b-0ce563-openshift-default-f75cc7fc9-445r7\" (UID: \"0a5f5114-bc2f-48c9-af68-f2d981ec9ce5\") " pod="openshift-ingress/e2e-api-b-0ce563-openshift-default-f75cc7fc9-445r7"
Oct 30 22:15:16.464669 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:16.464651    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a5f5114-bc2f-48c9-af68-f2d981ec9ce5-istiod-ca-cert\") pod \"e2e-api-b-0ce563-openshift-default-f75cc7fc9-445r7\" (UID: \"0a5f5114-bc2f-48c9-af68-f2d981ec9ce5\") " pod="openshift-ingress/e2e-api-b-0ce563-openshift-default-f75cc7fc9-445r7"
Oct 30 22:15:16.466257 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:16.466233    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a5f5114-bc2f-48c9-af68-f2d981ec9ce5-istio-envoy\") pod \"e2e-api-b-0ce563-openshift-default-f75cc7fc9-445r7\" (UID: \"0a5f5114-bc2f-48c9-af68-f2d981ec9ce5\") " pod="openshift-ingress/e2e-api-b-0ce563-openshift-default-f75cc7fc9-445r7"
Oct 30 22:15:16.466767 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:16.466747    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a5f5114-bc2f-48c9-af68-f2d981ec9ce5-istio-podinfo\") pod \"e2e-api-b-0ce563-openshift-default-f75cc7fc9-445r7\" (UID: \"0a5f5114-bc2f-48c9-af68-f2d981ec9ce5\") " pod="openshift-ingress/e2e-api-b-0ce563-openshift-default-f75cc7fc9-445r7"
Oct 30 22:15:16.520270 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:16.520176    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5swlb\" (UniqueName: \"kubernetes.io/projected/0a5f5114-bc2f-48c9-af68-f2d981ec9ce5-kube-api-access-5swlb\") pod \"e2e-api-b-0ce563-openshift-default-f75cc7fc9-445r7\" (UID: \"0a5f5114-bc2f-48c9-af68-f2d981ec9ce5\") " pod="openshift-ingress/e2e-api-b-0ce563-openshift-default-f75cc7fc9-445r7"
Oct 30 22:15:16.520911 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:16.520874    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a5f5114-bc2f-48c9-af68-f2d981ec9ce5-istio-token\") pod \"e2e-api-b-0ce563-openshift-default-f75cc7fc9-445r7\" (UID: \"0a5f5114-bc2f-48c9-af68-f2d981ec9ce5\") " pod="openshift-ingress/e2e-api-b-0ce563-openshift-default-f75cc7fc9-445r7"
Oct 30 22:15:16.527111 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:16.527088    5061 scope.go:122] "RemoveContainer" containerID="04e15d99ade3218213acf362ff00652471c87c00ca5e1c6009acef8b49ea46c8"
Oct 30 22:15:16.527678 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:16.527635    5061 scope.go:122] "RemoveContainer" containerID="09a0f8e9190d2cfa49446ad8a6c2c9b0e5f26e73843fb649cf6c592faeec3f05"
Oct 30 22:15:16.527808 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:16.527741    5061 scope.go:122] "RemoveContainer" containerID="844a1473d20b228d7e2bae4c60e35dff3bc384aa51536a79cb8f3c8527fbbb75"
Oct 30 22:15:16.527808 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:16.527766    5061 scope.go:122] "RemoveContainer" containerID="7459dbb94cf3d411470df36dcba68ec8b82760981fd11d816dbfe6163c18b5da"
Oct 30 22:15:16.527925 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:16.527901    5061 scope.go:122] "RemoveContainer" containerID="847811d9d3d28d49a7bae39801791c1de8cf92f8506dca3ea4bfe8cb2fa90c1f"
Oct 30 22:15:16.583088 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:16.583060    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-0ce563-openshift-default-f75cc7fc9-445r7"
Oct 30 22:15:17.072484 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:17.072451    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-0ce563-openshift-default-f75cc7fc9-445r7"]
Oct 30 22:15:17.073178 ip-10-0-130-60 kubenswrapper[5061]: W1030 22:15:17.072859    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a5f5114_bc2f_48c9_af68_f2d981ec9ce5.slice/crio-5e84611aabf2ec82c19d681e131825133046bffd7a6207e0c51849dc3ecb149b WatchSource:0}: Error finding container 5e84611aabf2ec82c19d681e131825133046bffd7a6207e0c51849dc3ecb149b: Status 404 returned error can't find the container with id 5e84611aabf2ec82c19d681e131825133046bffd7a6207e0c51849dc3ecb149b
Oct 30 22:15:17.074893 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:17.074858    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:15:17.074976 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:17.074923    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:15:17.074976 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:17.074953    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:15:17.110854 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:17.110831    5061 generic.go:356] "Generic (PLEG): container finished" podID="f220141f-4150-4f9c-9e14-631777d0ac14" containerID="07fdf70a88a04ac7efc92c5e49d0ebd555edfb6f1566d39343b15b8293059b8c" exitCode=60
Oct 30 22:15:17.110946 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:17.110910    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-dq9xr" event={"ID":"f220141f-4150-4f9c-9e14-631777d0ac14","Type":"ContainerDied","Data":"07fdf70a88a04ac7efc92c5e49d0ebd555edfb6f1566d39343b15b8293059b8c"}
Oct 30 22:15:17.110996 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:17.110973    5061 scope.go:122] "RemoveContainer" containerID="087ef8ed22156e9fa524a056c90869a98bab01ec17606047bd67904711618db5"
Oct 30 22:15:17.111886 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:17.111866    5061 scope.go:122] "RemoveContainer" containerID="07fdf70a88a04ac7efc92c5e49d0ebd555edfb6f1566d39343b15b8293059b8c"
Oct 30 22:15:17.112265 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:15:17.112242    5061 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889975-dq9xr_odh-ai-gateway-infra(f220141f-4150-4f9c-9e14-631777d0ac14)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-dq9xr" podUID="f220141f-4150-4f9c-9e14-631777d0ac14"
Oct 30 22:15:17.134256 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:17.134215    5061 generic.go:356] "Generic (PLEG): container finished" podID="8b4f58e4-2611-45f1-b431-a33ce392d964" containerID="4b9f57dc4c1733ce47e979b2d62225e7a7913ac075487123a961cfd2366c525f" exitCode=60
Oct 30 22:15:17.134470 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:17.134379    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e66edb-29889975-flzf6" event={"ID":"8b4f58e4-2611-45f1-b431-a33ce392d964","Type":"ContainerDied","Data":"4b9f57dc4c1733ce47e979b2d62225e7a7913ac075487123a961cfd2366c525f"}
Oct 30 22:15:17.134470 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:17.134428    5061 scope.go:122] "RemoveContainer" containerID="fb3dc4c5756703c6bf6bb73c15a55aee39110ba2c19f6c8536de6603631aaec7"
Oct 30 22:15:17.134885 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:17.134868    5061 scope.go:122] "RemoveContainer" containerID="4b9f57dc4c1733ce47e979b2d62225e7a7913ac075487123a961cfd2366c525f"
Oct 30 22:15:17.135210 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:15:17.135190    5061 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-e66edb-29889975-flzf6_odh-ai-gateway-infra(8b4f58e4-2611-45f1-b431-a33ce392d964)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e66edb-29889975-flzf6" podUID="8b4f58e4-2611-45f1-b431-a33ce392d964"
Oct 30 22:15:17.137889 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:17.137561    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0ce563-openshift-default-f75cc7fc9-445r7" event={"ID":"0a5f5114-bc2f-48c9-af68-f2d981ec9ce5","Type":"ContainerStarted","Data":"5e84611aabf2ec82c19d681e131825133046bffd7a6207e0c51849dc3ecb149b"}
Oct 30 22:15:17.228230 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:17.228175    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889975-dq9xr"]
Oct 30 22:15:17.336005 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:17.335969    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e66edb-29889975-flzf6"]
Oct 30 22:15:17.521267 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:17.521192    5061 scope.go:122] "RemoveContainer" containerID="4e661358350ceeaed21dd29b7429217514d11f83cb0e55a4664963b8fcf1a144"
Oct 30 22:15:17.521444 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:17.521343    5061 scope.go:122] "RemoveContainer" containerID="31464249c4951adb2f8ba7b6f6eec05cc32771225c8cc082b82dee5caeec4f25"
Oct 30 22:15:17.521507 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:17.521475    5061 scope.go:122] "RemoveContainer" containerID="82dddbddcfcce85bc3d85b3bf22ef211a56c6bc33ad80549fee2923e0cdf341b"
Oct 30 22:15:17.619438 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:17.619410    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-778d39d3-d4gvh"]
Oct 30 22:15:17.625927 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:17.625906    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-778d39d3-d4gvh"
Oct 30 22:15:17.675897 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:17.675869    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:15:17.682719 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:17.680386    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-778d39d3-d4gvh"]
Oct 30 22:15:17.784201 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:17.784138    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bxh5\" (UniqueName: \"kubernetes.io/projected/2e7528ce-43ab-4e4c-be3f-b5a87b82d3b5-kube-api-access-9bxh5\") pod \"maas-api-revoke-keys-e2e-mt-778d39d3-d4gvh\" (UID: \"2e7528ce-43ab-4e4c-be3f-b5a87b82d3b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-778d39d3-d4gvh"
Oct 30 22:15:17.784327 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:17.784303    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e7528ce-43ab-4e4c-be3f-b5a87b82d3b5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-778d39d3-d4gvh\" (UID: \"2e7528ce-43ab-4e4c-be3f-b5a87b82d3b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-778d39d3-d4gvh"
Oct 30 22:15:17.784441 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:17.784412    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e7528ce-43ab-4e4c-be3f-b5a87b82d3b5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-778d39d3-d4gvh\" (UID: \"2e7528ce-43ab-4e4c-be3f-b5a87b82d3b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-778d39d3-d4gvh"
Oct 30 22:15:17.886223 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:17.886195    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bxh5\" (UniqueName: \"kubernetes.io/projected/2e7528ce-43ab-4e4c-be3f-b5a87b82d3b5-kube-api-access-9bxh5\") pod \"maas-api-revoke-keys-e2e-mt-778d39d3-d4gvh\" (UID: \"2e7528ce-43ab-4e4c-be3f-b5a87b82d3b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-778d39d3-d4gvh"
Oct 30 22:15:17.886370 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:17.886353    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e7528ce-43ab-4e4c-be3f-b5a87b82d3b5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-778d39d3-d4gvh\" (UID: \"2e7528ce-43ab-4e4c-be3f-b5a87b82d3b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-778d39d3-d4gvh"
Oct 30 22:15:17.886487 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:17.886472    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e7528ce-43ab-4e4c-be3f-b5a87b82d3b5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-778d39d3-d4gvh\" (UID: \"2e7528ce-43ab-4e4c-be3f-b5a87b82d3b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-778d39d3-d4gvh"
Oct 30 22:15:17.886998 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:17.886976    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e7528ce-43ab-4e4c-be3f-b5a87b82d3b5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-778d39d3-d4gvh\" (UID: \"2e7528ce-43ab-4e4c-be3f-b5a87b82d3b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-778d39d3-d4gvh"
Oct 30 22:15:17.888735 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:17.888715    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e7528ce-43ab-4e4c-be3f-b5a87b82d3b5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-778d39d3-d4gvh\" (UID: \"2e7528ce-43ab-4e4c-be3f-b5a87b82d3b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-778d39d3-d4gvh"
Oct 30 22:15:17.940637 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:17.940611    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bxh5\" (UniqueName: \"kubernetes.io/projected/2e7528ce-43ab-4e4c-be3f-b5a87b82d3b5-kube-api-access-9bxh5\") pod \"maas-api-revoke-keys-e2e-mt-778d39d3-d4gvh\" (UID: \"2e7528ce-43ab-4e4c-be3f-b5a87b82d3b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-778d39d3-d4gvh"
Oct 30 22:15:17.942481 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:17.942460    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-778d39d3-d4gvh"
Oct 30 22:15:18.151489 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:18.151455    5061 generic.go:356] "Generic (PLEG): container finished" podID="42adc96b-1b21-484e-b6fa-51bbf1127ac0" containerID="eaeae843ba7656c9a28037048ab31364e8efec6a884f7468f2ddd1c62e3136f1" exitCode=60
Oct 30 22:15:18.151698 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:18.151590    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-4d8041-29889975-fn5gn" event={"ID":"42adc96b-1b21-484e-b6fa-51bbf1127ac0","Type":"ContainerDied","Data":"eaeae843ba7656c9a28037048ab31364e8efec6a884f7468f2ddd1c62e3136f1"}
Oct 30 22:15:18.151698 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:18.151666    5061 scope.go:122] "RemoveContainer" containerID="7459dbb94cf3d411470df36dcba68ec8b82760981fd11d816dbfe6163c18b5da"
Oct 30 22:15:18.152154 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:18.152133    5061 scope.go:122] "RemoveContainer" containerID="eaeae843ba7656c9a28037048ab31364e8efec6a884f7468f2ddd1c62e3136f1"
Oct 30 22:15:18.152852 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:15:18.152822    5061 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-4d8041-29889975-fn5gn_odh-ai-gateway-infra(42adc96b-1b21-484e-b6fa-51bbf1127ac0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-4d8041-29889975-fn5gn" podUID="42adc96b-1b21-484e-b6fa-51bbf1127ac0"
Oct 30 22:15:18.155935 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:18.155902    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0ce563-openshift-default-f75cc7fc9-445r7" event={"ID":"0a5f5114-bc2f-48c9-af68-f2d981ec9ce5","Type":"ContainerStarted","Data":"c26f9976807884d9967ba246acee4e31c5f8093e873cdd5407d6f8f3cde6f64d"}
Oct 30 22:15:18.159154 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:18.158729    5061 generic.go:356] "Generic (PLEG): container finished" podID="3787aeb2-4e6e-4855-8be8-137ebcd39a26" containerID="0b2dde3bc4be057d8f2e457b1fe85cbf4cb48e6903c66b54dfc322db1064ee02" exitCode=60
Oct 30 22:15:18.159154 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:18.158827    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-778d39d3-29889975-mct28" event={"ID":"3787aeb2-4e6e-4855-8be8-137ebcd39a26","Type":"ContainerDied","Data":"0b2dde3bc4be057d8f2e457b1fe85cbf4cb48e6903c66b54dfc322db1064ee02"}
Oct 30 22:15:18.159430 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:18.159405    5061 scope.go:122] "RemoveContainer" containerID="0b2dde3bc4be057d8f2e457b1fe85cbf4cb48e6903c66b54dfc322db1064ee02"
Oct 30 22:15:18.159908 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:15:18.159871    5061 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-778d39d3-29889975-mct28_odh-ai-gateway-infra(3787aeb2-4e6e-4855-8be8-137ebcd39a26)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-778d39d3-29889975-mct28" podUID="3787aeb2-4e6e-4855-8be8-137ebcd39a26"
Oct 30 22:15:18.162121 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:18.162089    5061 generic.go:356] "Generic (PLEG): container finished" podID="06e58c0b-aa5c-4f1e-8228-7070a03657bb" containerID="c6360753e6f88eb6cd0d0b3b68a9d7b3a6229d379fb03b86e5813ca18544f445" exitCode=60
Oct 30 22:15:18.162212 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:18.162138    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-02bd3b-29889975-jjfj2" event={"ID":"06e58c0b-aa5c-4f1e-8228-7070a03657bb","Type":"ContainerDied","Data":"c6360753e6f88eb6cd0d0b3b68a9d7b3a6229d379fb03b86e5813ca18544f445"}
Oct 30 22:15:18.162909 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:18.162811    5061 scope.go:122] "RemoveContainer" containerID="c6360753e6f88eb6cd0d0b3b68a9d7b3a6229d379fb03b86e5813ca18544f445"
Oct 30 22:15:18.163199 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:15:18.163162    5061 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-02bd3b-29889975-jjfj2_odh-ai-gateway-infra(06e58c0b-aa5c-4f1e-8228-7070a03657bb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-02bd3b-29889975-jjfj2" podUID="06e58c0b-aa5c-4f1e-8228-7070a03657bb"
Oct 30 22:15:18.172495 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:18.172468    5061 scope.go:122] "RemoveContainer" containerID="847811d9d3d28d49a7bae39801791c1de8cf92f8506dca3ea4bfe8cb2fa90c1f"
Oct 30 22:15:18.173196 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:18.173165    5061 generic.go:356] "Generic (PLEG): container finished" podID="0ba41c1f-1fc3-4a32-967c-2d86878e2887" containerID="a31b3bea1a44d9a45d4e8fe0aa6c1588b8f5dc76a564fc9cb90ef66e2d1c3770" exitCode=60
Oct 30 22:15:18.173318 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:18.173253    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5402dd-29889975-jjcmb" event={"ID":"0ba41c1f-1fc3-4a32-967c-2d86878e2887","Type":"ContainerDied","Data":"a31b3bea1a44d9a45d4e8fe0aa6c1588b8f5dc76a564fc9cb90ef66e2d1c3770"}
Oct 30 22:15:18.177568 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:18.174518    5061 scope.go:122] "RemoveContainer" containerID="a31b3bea1a44d9a45d4e8fe0aa6c1588b8f5dc76a564fc9cb90ef66e2d1c3770"
Oct 30 22:15:18.177568 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:15:18.175677    5061 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-5402dd-29889975-jjcmb_odh-ai-gateway-infra(0ba41c1f-1fc3-4a32-967c-2d86878e2887)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5402dd-29889975-jjcmb" podUID="0ba41c1f-1fc3-4a32-967c-2d86878e2887"
Oct 30 22:15:18.180311 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:18.180285    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-6ea8da-29889975-r8jsf" event={"ID":"08aac4a2-7d84-4b74-bf47-dc849fd6e446","Type":"ContainerStarted","Data":"a811babd464d25f7342e6f44d615cec374723f76293f8e44ec1d72a8794e45f9"}
Oct 30 22:15:18.182481 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:18.182460    5061 generic.go:356] "Generic (PLEG): container finished" podID="777f3189-c018-4cdb-92ba-eeb10b71e868" containerID="2009ea126331a2f9681f403a8cb2cffde983feed727bc307db936b4b3d605b7e" exitCode=60
Oct 30 22:15:18.182583 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:18.182521    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-25de17-29889975-qk5j5" event={"ID":"777f3189-c018-4cdb-92ba-eeb10b71e868","Type":"ContainerDied","Data":"2009ea126331a2f9681f403a8cb2cffde983feed727bc307db936b4b3d605b7e"}
Oct 30 22:15:18.182919 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:18.182904    5061 scope.go:122] "RemoveContainer" containerID="2009ea126331a2f9681f403a8cb2cffde983feed727bc307db936b4b3d605b7e"
Oct 30 22:15:18.183209 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:15:18.183182    5061 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-25de17-29889975-qk5j5_odh-ai-gateway-infra(777f3189-c018-4cdb-92ba-eeb10b71e868)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-25de17-29889975-qk5j5" podUID="777f3189-c018-4cdb-92ba-eeb10b71e868"
Oct 30 22:15:18.192866 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:18.192847    5061 scope.go:122] "RemoveContainer" containerID="09a0f8e9190d2cfa49446ad8a6c2c9b0e5f26e73843fb649cf6c592faeec3f05"
Oct 30 22:15:18.206827 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:18.206805    5061 scope.go:122] "RemoveContainer" containerID="844a1473d20b228d7e2bae4c60e35dff3bc384aa51536a79cb8f3c8527fbbb75"
Oct 30 22:15:18.224432 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:18.224411    5061 scope.go:122] "RemoveContainer" containerID="04e15d99ade3218213acf362ff00652471c87c00ca5e1c6009acef8b49ea46c8"
Oct 30 22:15:18.259352 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:18.259320    5061 patch_prober.go:29] interesting pod/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-n4bnf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:15:18.260591 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:18.259374    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-n4bnf" podUID="799953d1-1dc8-421e-9e98-d2696d399ac2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:15:18.322971 ip-10-0-130-60 kubenswrapper[5061]: W1030 22:15:18.322926    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e7528ce_43ab_4e4c_be3f_b5a87b82d3b5.slice/crio-b9287ce9485abd742af383e54abb52ec5efb8a20d616156d00c5b31048e7ad41 WatchSource:0}: Error finding container b9287ce9485abd742af383e54abb52ec5efb8a20d616156d00c5b31048e7ad41: Status 404 returned error can't find the container with id b9287ce9485abd742af383e54abb52ec5efb8a20d616156d00c5b31048e7ad41
Oct 30 22:15:18.334190 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:18.328889    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-778d39d3-d4gvh"]
Oct 30 22:15:18.430953 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:18.430927    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e66edb-29889975-flzf6"
Oct 30 22:15:18.484499 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:18.484472    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-dq9xr"
Oct 30 22:15:18.552245 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:18.552148    5061 patch_prober.go:29] interesting pod/e2e-worker-w3-f776cb-openshift-default-74d577586f-5z645 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:15:18.552245 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:18.552193    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-5z645" podUID="a84e4890-ba61-49a5-b9b6-be9cf741d2d8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:15:18.583665 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:18.583633    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-0ce563-openshift-default-f75cc7fc9-445r7"
Oct 30 22:15:18.589487 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:18.589462    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-0ce563-openshift-default-f75cc7fc9-445r7"
Oct 30 22:15:18.604915 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:18.604881    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f220141f-4150-4f9c-9e14-631777d0ac14-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f220141f-4150-4f9c-9e14-631777d0ac14-maas-api-service-ca\") pod \"f220141f-4150-4f9c-9e14-631777d0ac14\" (UID: \"f220141f-4150-4f9c-9e14-631777d0ac14\") "
Oct 30 22:15:18.605046 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:18.604998    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b4f58e4-2611-45f1-b431-a33ce392d964-kube-api-access-gbf6h\" (UniqueName: \"kubernetes.io/projected/8b4f58e4-2611-45f1-b431-a33ce392d964-kube-api-access-gbf6h\") pod \"8b4f58e4-2611-45f1-b431-a33ce392d964\" (UID: \"8b4f58e4-2611-45f1-b431-a33ce392d964\") "
Oct 30 22:15:18.605107 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:18.605087    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b4f58e4-2611-45f1-b431-a33ce392d964-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b4f58e4-2611-45f1-b431-a33ce392d964-maas-api-service-ca\") pod \"8b4f58e4-2611-45f1-b431-a33ce392d964\" (UID: \"8b4f58e4-2611-45f1-b431-a33ce392d964\") "
Oct 30 22:15:18.605240 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:18.605224    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f220141f-4150-4f9c-9e14-631777d0ac14-kube-api-access-dwjn9\" (UniqueName: \"kubernetes.io/projected/f220141f-4150-4f9c-9e14-631777d0ac14-kube-api-access-dwjn9\") pod \"f220141f-4150-4f9c-9e14-631777d0ac14\" (UID: \"f220141f-4150-4f9c-9e14-631777d0ac14\") "
Oct 30 22:15:18.605390 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:18.605364    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f220141f-4150-4f9c-9e14-631777d0ac14-maas-api-service-ca" pod "f220141f-4150-4f9c-9e14-631777d0ac14" (UID: "f220141f-4150-4f9c-9e14-631777d0ac14"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:15:18.605455 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:18.605416    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b4f58e4-2611-45f1-b431-a33ce392d964-maas-api-service-ca" pod "8b4f58e4-2611-45f1-b431-a33ce392d964" (UID: "8b4f58e4-2611-45f1-b431-a33ce392d964"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:15:18.605835 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:18.605815    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f220141f-4150-4f9c-9e14-631777d0ac14-maas-api-service-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:18.605927 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:18.605838    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b4f58e4-2611-45f1-b431-a33ce392d964-maas-api-service-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:18.606118 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:18.606079    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-0ce563-openshift-default-f75cc7fc9-445r7" podStartSLOduration=2.606065971 podStartE2EDuration="2.606065971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:18.602763884 +0000 UTC m=+1972.215533642" watchObservedRunningTime="2026-10-30 22:15:18.606065971 +0000 UTC m=+1972.218835728"
Oct 30 22:15:18.610294 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:18.610140    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f220141f-4150-4f9c-9e14-631777d0ac14-kube-api-access-dwjn9" pod "f220141f-4150-4f9c-9e14-631777d0ac14" (UID: "f220141f-4150-4f9c-9e14-631777d0ac14"). InnerVolumeSpecName "kube-api-access-dwjn9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:18.611148 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:18.611122    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b4f58e4-2611-45f1-b431-a33ce392d964-kube-api-access-gbf6h" pod "8b4f58e4-2611-45f1-b431-a33ce392d964" (UID: "8b4f58e4-2611-45f1-b431-a33ce392d964"). InnerVolumeSpecName "kube-api-access-gbf6h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:18.707184 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:18.707152    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbf6h\" (UniqueName: \"kubernetes.io/projected/8b4f58e4-2611-45f1-b431-a33ce392d964-kube-api-access-gbf6h\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:18.707184 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:18.707181    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwjn9\" (UniqueName: \"kubernetes.io/projected/f220141f-4150-4f9c-9e14-631777d0ac14-kube-api-access-dwjn9\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:18.738690 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:18.719290    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-n4bnf"
Oct 30 22:15:18.807881 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:18.807843    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/799953d1-1dc8-421e-9e98-d2696d399ac2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/799953d1-1dc8-421e-9e98-d2696d399ac2-workload-socket\") pod \"799953d1-1dc8-421e-9e98-d2696d399ac2\" (UID: \"799953d1-1dc8-421e-9e98-d2696d399ac2\") "
Oct 30 22:15:18.808057 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:18.807900    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/799953d1-1dc8-421e-9e98-d2696d399ac2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/799953d1-1dc8-421e-9e98-d2696d399ac2-istio-podinfo\") pod \"799953d1-1dc8-421e-9e98-d2696d399ac2\" (UID: \"799953d1-1dc8-421e-9e98-d2696d399ac2\") "
Oct 30 22:15:18.808057 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:18.807998    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/799953d1-1dc8-421e-9e98-d2696d399ac2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/799953d1-1dc8-421e-9e98-d2696d399ac2-istio-envoy\") pod \"799953d1-1dc8-421e-9e98-d2696d399ac2\" (UID: \"799953d1-1dc8-421e-9e98-d2696d399ac2\") "
Oct 30 22:15:18.808057 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:18.808048    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/799953d1-1dc8-421e-9e98-d2696d399ac2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/799953d1-1dc8-421e-9e98-d2696d399ac2-istio-data\") pod \"799953d1-1dc8-421e-9e98-d2696d399ac2\" (UID: \"799953d1-1dc8-421e-9e98-d2696d399ac2\") "
Oct 30 22:15:18.808185 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:18.808081    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/799953d1-1dc8-421e-9e98-d2696d399ac2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/799953d1-1dc8-421e-9e98-d2696d399ac2-istiod-ca-cert\") pod \"799953d1-1dc8-421e-9e98-d2696d399ac2\" (UID: \"799953d1-1dc8-421e-9e98-d2696d399ac2\") "
Oct 30 22:15:18.808185 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:18.808122    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/799953d1-1dc8-421e-9e98-d2696d399ac2-kube-api-access-gjm2l\" (UniqueName: \"kubernetes.io/projected/799953d1-1dc8-421e-9e98-d2696d399ac2-kube-api-access-gjm2l\") pod \"799953d1-1dc8-421e-9e98-d2696d399ac2\" (UID: \"799953d1-1dc8-421e-9e98-d2696d399ac2\") "
Oct 30 22:15:18.808284 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:18.808256    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/799953d1-1dc8-421e-9e98-d2696d399ac2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/799953d1-1dc8-421e-9e98-d2696d399ac2-credential-socket\") pod \"799953d1-1dc8-421e-9e98-d2696d399ac2\" (UID: \"799953d1-1dc8-421e-9e98-d2696d399ac2\") "
Oct 30 22:15:18.808343 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:18.808327    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/799953d1-1dc8-421e-9e98-d2696d399ac2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/799953d1-1dc8-421e-9e98-d2696d399ac2-workload-certs\") pod \"799953d1-1dc8-421e-9e98-d2696d399ac2\" (UID: \"799953d1-1dc8-421e-9e98-d2696d399ac2\") "
Oct 30 22:15:18.808604 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:18.808464    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/799953d1-1dc8-421e-9e98-d2696d399ac2-istio-token\" (UniqueName: \"kubernetes.io/projected/799953d1-1dc8-421e-9e98-d2696d399ac2-istio-token\") pod \"799953d1-1dc8-421e-9e98-d2696d399ac2\" (UID: \"799953d1-1dc8-421e-9e98-d2696d399ac2\") "
Oct 30 22:15:18.808604 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:18.808517    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/799953d1-1dc8-421e-9e98-d2696d399ac2-istiod-ca-cert" pod "799953d1-1dc8-421e-9e98-d2696d399ac2" (UID: "799953d1-1dc8-421e-9e98-d2696d399ac2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:15:18.808776 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:18.808736    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/799953d1-1dc8-421e-9e98-d2696d399ac2-workload-certs" pod "799953d1-1dc8-421e-9e98-d2696d399ac2" (UID: "799953d1-1dc8-421e-9e98-d2696d399ac2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:15:18.808845 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:18.808767    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/799953d1-1dc8-421e-9e98-d2696d399ac2-workload-socket" pod "799953d1-1dc8-421e-9e98-d2696d399ac2" (UID: "799953d1-1dc8-421e-9e98-d2696d399ac2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:15:18.808909 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:18.808888    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/799953d1-1dc8-421e-9e98-d2696d399ac2-credential-socket" pod "799953d1-1dc8-421e-9e98-d2696d399ac2" (UID: "799953d1-1dc8-421e-9e98-d2696d399ac2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:15:18.810411 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:18.809002    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/799953d1-1dc8-421e-9e98-d2696d399ac2-istio-data" pod "799953d1-1dc8-421e-9e98-d2696d399ac2" (UID: "799953d1-1dc8-421e-9e98-d2696d399ac2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:15:18.810411 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:18.809317    5061 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/799953d1-1dc8-421e-9e98-d2696d399ac2-workload-certs\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:18.810411 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:18.809351    5061 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/799953d1-1dc8-421e-9e98-d2696d399ac2-workload-socket\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:18.810411 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:18.809366    5061 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/799953d1-1dc8-421e-9e98-d2696d399ac2-istio-data\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:18.810411 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:18.809381    5061 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/799953d1-1dc8-421e-9e98-d2696d399ac2-istiod-ca-cert\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:18.810411 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:18.809393    5061 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/799953d1-1dc8-421e-9e98-d2696d399ac2-credential-socket\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:18.811951 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:18.811891    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/799953d1-1dc8-421e-9e98-d2696d399ac2-istio-envoy" pod "799953d1-1dc8-421e-9e98-d2696d399ac2" (UID: "799953d1-1dc8-421e-9e98-d2696d399ac2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:15:18.811951 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:18.811908    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/799953d1-1dc8-421e-9e98-d2696d399ac2-kube-api-access-gjm2l" pod "799953d1-1dc8-421e-9e98-d2696d399ac2" (UID: "799953d1-1dc8-421e-9e98-d2696d399ac2"). InnerVolumeSpecName "kube-api-access-gjm2l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:18.812088 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:18.812005    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/799953d1-1dc8-421e-9e98-d2696d399ac2-istio-token" pod "799953d1-1dc8-421e-9e98-d2696d399ac2" (UID: "799953d1-1dc8-421e-9e98-d2696d399ac2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:18.813300 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:18.813277    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/799953d1-1dc8-421e-9e98-d2696d399ac2-istio-podinfo" pod "799953d1-1dc8-421e-9e98-d2696d399ac2" (UID: "799953d1-1dc8-421e-9e98-d2696d399ac2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:15:18.911329 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:18.911302    5061 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/799953d1-1dc8-421e-9e98-d2696d399ac2-istio-envoy\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:18.911329 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:18.911332    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjm2l\" (UniqueName: \"kubernetes.io/projected/799953d1-1dc8-421e-9e98-d2696d399ac2-kube-api-access-gjm2l\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:18.911572 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:18.911348    5061 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/799953d1-1dc8-421e-9e98-d2696d399ac2-istio-token\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:18.911572 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:18.911361    5061 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/799953d1-1dc8-421e-9e98-d2696d399ac2-istio-podinfo\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:19.108009 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:19.107978    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-5z645"
Oct 30 22:15:19.195192 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:19.195095    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-778d39d3-d4gvh" event={"ID":"2e7528ce-43ab-4e4c-be3f-b5a87b82d3b5","Type":"ContainerStarted","Data":"9db031599672d729bea66c018b477c94a91be9a60623547868bb62fc1213d37a"}
Oct 30 22:15:19.195341 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:19.195191    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-778d39d3-d4gvh" event={"ID":"2e7528ce-43ab-4e4c-be3f-b5a87b82d3b5","Type":"ContainerStarted","Data":"b9287ce9485abd742af383e54abb52ec5efb8a20d616156d00c5b31048e7ad41"}
Oct 30 22:15:19.196432 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:19.196413    5061 generic.go:356] "Generic (PLEG): container finished" podID="a84e4890-ba61-49a5-b9b6-be9cf741d2d8" containerID="b4886f646e7e94afb16aff7aa126959dd64ed13c753ca92a3013d38214007232" exitCode=0
Oct 30 22:15:19.196579 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:19.196463    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-5z645"
Oct 30 22:15:19.196579 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:19.196480    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-5z645" event={"ID":"a84e4890-ba61-49a5-b9b6-be9cf741d2d8","Type":"ContainerDied","Data":"b4886f646e7e94afb16aff7aa126959dd64ed13c753ca92a3013d38214007232"}
Oct 30 22:15:19.196579 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:19.196512    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-5z645" event={"ID":"a84e4890-ba61-49a5-b9b6-be9cf741d2d8","Type":"ContainerDied","Data":"834b6990ef43d6b1085f57128fcb5f4446c48d344d632e8bea6abd57f92d24b7"}
Oct 30 22:15:19.196579 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:19.196555    5061 scope.go:122] "RemoveContainer" containerID="b4886f646e7e94afb16aff7aa126959dd64ed13c753ca92a3013d38214007232"
Oct 30 22:15:19.198419 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:19.198387    5061 generic.go:356] "Generic (PLEG): container finished" podID="52312d9d-e2f8-4ad3-bce1-53ffc5b08d4f" containerID="49e84ec08d7ae6bfe29d611f8bbe53eb6a64b6bfed0e377db6cd8b48a1555a04" exitCode=60
Oct 30 22:15:19.198522 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:19.198442    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-411ff1-29889975-fc8l4" event={"ID":"52312d9d-e2f8-4ad3-bce1-53ffc5b08d4f","Type":"ContainerDied","Data":"49e84ec08d7ae6bfe29d611f8bbe53eb6a64b6bfed0e377db6cd8b48a1555a04"}
Oct 30 22:15:19.198890 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:19.198872    5061 scope.go:122] "RemoveContainer" containerID="49e84ec08d7ae6bfe29d611f8bbe53eb6a64b6bfed0e377db6cd8b48a1555a04"
Oct 30 22:15:19.199203 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:15:19.199184    5061 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-411ff1-29889975-fc8l4_odh-ai-gateway-infra(52312d9d-e2f8-4ad3-bce1-53ffc5b08d4f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-411ff1-29889975-fc8l4" podUID="52312d9d-e2f8-4ad3-bce1-53ffc5b08d4f"
Oct 30 22:15:19.199905 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:19.199890    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-dq9xr"
Oct 30 22:15:19.200015 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:19.199936    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-dq9xr" event={"ID":"f220141f-4150-4f9c-9e14-631777d0ac14","Type":"ContainerDied","Data":"74ea822855504dd1f4fc36a274b5be614dcac462259ee2582463d739c242d447"}
Oct 30 22:15:19.201392 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:19.201364    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e66edb-29889975-flzf6" event={"ID":"8b4f58e4-2611-45f1-b431-a33ce392d964","Type":"ContainerDied","Data":"010d1790a0b89cf36b7c5a2c6dd7d94d05ff7a79caf7401a4fd26c5fe49a8e64"}
Oct 30 22:15:19.201392 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:19.201383    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e66edb-29889975-flzf6"
Oct 30 22:15:19.208654 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:19.208630    5061 scope.go:122] "RemoveContainer" containerID="b4886f646e7e94afb16aff7aa126959dd64ed13c753ca92a3013d38214007232"
Oct 30 22:15:19.209065 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:15:19.208956    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4886f646e7e94afb16aff7aa126959dd64ed13c753ca92a3013d38214007232\": container with ID starting with b4886f646e7e94afb16aff7aa126959dd64ed13c753ca92a3013d38214007232 not found: ID does not exist" containerID="b4886f646e7e94afb16aff7aa126959dd64ed13c753ca92a3013d38214007232"
Oct 30 22:15:19.209065 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:19.208990    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4886f646e7e94afb16aff7aa126959dd64ed13c753ca92a3013d38214007232"} err="failed to get container status \"b4886f646e7e94afb16aff7aa126959dd64ed13c753ca92a3013d38214007232\": rpc error: code = NotFound desc = could not find container \"b4886f646e7e94afb16aff7aa126959dd64ed13c753ca92a3013d38214007232\": container with ID starting with b4886f646e7e94afb16aff7aa126959dd64ed13c753ca92a3013d38214007232 not found: ID does not exist"
Oct 30 22:15:19.209065 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:19.209011    5061 scope.go:122] "RemoveContainer" containerID="4e661358350ceeaed21dd29b7429217514d11f83cb0e55a4664963b8fcf1a144"
Oct 30 22:15:19.212429 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:19.212406    5061 generic.go:356] "Generic (PLEG): container finished" podID="08aac4a2-7d84-4b74-bf47-dc849fd6e446" containerID="a811babd464d25f7342e6f44d615cec374723f76293f8e44ec1d72a8794e45f9" exitCode=60
Oct 30 22:15:19.212569 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:19.212469    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-6ea8da-29889975-r8jsf" event={"ID":"08aac4a2-7d84-4b74-bf47-dc849fd6e446","Type":"ContainerDied","Data":"a811babd464d25f7342e6f44d615cec374723f76293f8e44ec1d72a8794e45f9"}
Oct 30 22:15:19.212872 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:19.212847    5061 scope.go:122] "RemoveContainer" containerID="a811babd464d25f7342e6f44d615cec374723f76293f8e44ec1d72a8794e45f9"
Oct 30 22:15:19.213154 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:15:19.213129    5061 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-6ea8da-29889975-r8jsf_odh-ai-gateway-infra(08aac4a2-7d84-4b74-bf47-dc849fd6e446)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-6ea8da-29889975-r8jsf" podUID="08aac4a2-7d84-4b74-bf47-dc849fd6e446"
Oct 30 22:15:19.217836 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:19.217815    5061 generic.go:356] "Generic (PLEG): container finished" podID="cbaff831-8881-4762-8332-25c8eb4c2a28" containerID="cb32765cf6e669d9e8b83e6886e3f8e10b647297913e9ca34862d174f0b825e3" exitCode=60
Oct 30 22:15:19.217950 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:19.217884    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5fef81-29889975-fdwmx" event={"ID":"cbaff831-8881-4762-8332-25c8eb4c2a28","Type":"ContainerDied","Data":"cb32765cf6e669d9e8b83e6886e3f8e10b647297913e9ca34862d174f0b825e3"}
Oct 30 22:15:19.217950 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:19.217929    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a84e4890-ba61-49a5-b9b6-be9cf741d2d8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a84e4890-ba61-49a5-b9b6-be9cf741d2d8-istio-podinfo\") pod \"a84e4890-ba61-49a5-b9b6-be9cf741d2d8\" (UID: \"a84e4890-ba61-49a5-b9b6-be9cf741d2d8\") "
Oct 30 22:15:19.218068 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:19.217982    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a84e4890-ba61-49a5-b9b6-be9cf741d2d8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a84e4890-ba61-49a5-b9b6-be9cf741d2d8-credential-socket\") pod \"a84e4890-ba61-49a5-b9b6-be9cf741d2d8\" (UID: \"a84e4890-ba61-49a5-b9b6-be9cf741d2d8\") "
Oct 30 22:15:19.218118 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:19.218086    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a84e4890-ba61-49a5-b9b6-be9cf741d2d8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a84e4890-ba61-49a5-b9b6-be9cf741d2d8-istio-data\") pod \"a84e4890-ba61-49a5-b9b6-be9cf741d2d8\" (UID: \"a84e4890-ba61-49a5-b9b6-be9cf741d2d8\") "
Oct 30 22:15:19.218224 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:19.218204    5061 scope.go:122] "RemoveContainer" containerID="cb32765cf6e669d9e8b83e6886e3f8e10b647297913e9ca34862d174f0b825e3"
Oct 30 22:15:19.218331 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:19.218301    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a84e4890-ba61-49a5-b9b6-be9cf741d2d8-istio-data" pod "a84e4890-ba61-49a5-b9b6-be9cf741d2d8" (UID: "a84e4890-ba61-49a5-b9b6-be9cf741d2d8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:15:19.218331 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:19.218325    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a84e4890-ba61-49a5-b9b6-be9cf741d2d8-credential-socket" pod "a84e4890-ba61-49a5-b9b6-be9cf741d2d8" (UID: "a84e4890-ba61-49a5-b9b6-be9cf741d2d8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:15:19.218518 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:15:19.218497    5061 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-5fef81-29889975-fdwmx_odh-ai-gateway-infra(cbaff831-8881-4762-8332-25c8eb4c2a28)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5fef81-29889975-fdwmx" podUID="cbaff831-8881-4762-8332-25c8eb4c2a28"
Oct 30 22:15:19.218637 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:19.218499    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a84e4890-ba61-49a5-b9b6-be9cf741d2d8-istio-token\" (UniqueName: \"kubernetes.io/projected/a84e4890-ba61-49a5-b9b6-be9cf741d2d8-istio-token\") pod \"a84e4890-ba61-49a5-b9b6-be9cf741d2d8\" (UID: \"a84e4890-ba61-49a5-b9b6-be9cf741d2d8\") "
Oct 30 22:15:19.218703 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:19.218635    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a84e4890-ba61-49a5-b9b6-be9cf741d2d8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a84e4890-ba61-49a5-b9b6-be9cf741d2d8-istiod-ca-cert\") pod \"a84e4890-ba61-49a5-b9b6-be9cf741d2d8\" (UID: \"a84e4890-ba61-49a5-b9b6-be9cf741d2d8\") "
Oct 30 22:15:19.218761 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:19.218707    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a84e4890-ba61-49a5-b9b6-be9cf741d2d8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a84e4890-ba61-49a5-b9b6-be9cf741d2d8-workload-socket\") pod \"a84e4890-ba61-49a5-b9b6-be9cf741d2d8\" (UID: \"a84e4890-ba61-49a5-b9b6-be9cf741d2d8\") "
Oct 30 22:15:19.218824 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:19.218757    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a84e4890-ba61-49a5-b9b6-be9cf741d2d8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a84e4890-ba61-49a5-b9b6-be9cf741d2d8-istio-envoy\") pod \"a84e4890-ba61-49a5-b9b6-be9cf741d2d8\" (UID: \"a84e4890-ba61-49a5-b9b6-be9cf741d2d8\") "
Oct 30 22:15:19.218824 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:19.218788    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a84e4890-ba61-49a5-b9b6-be9cf741d2d8-kube-api-access-mfm9k\" (UniqueName: \"kubernetes.io/projected/a84e4890-ba61-49a5-b9b6-be9cf741d2d8-kube-api-access-mfm9k\") pod \"a84e4890-ba61-49a5-b9b6-be9cf741d2d8\" (UID: \"a84e4890-ba61-49a5-b9b6-be9cf741d2d8\") "
Oct 30 22:15:19.218929 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:19.218914    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a84e4890-ba61-49a5-b9b6-be9cf741d2d8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a84e4890-ba61-49a5-b9b6-be9cf741d2d8-workload-certs\") pod \"a84e4890-ba61-49a5-b9b6-be9cf741d2d8\" (UID: \"a84e4890-ba61-49a5-b9b6-be9cf741d2d8\") "
Oct 30 22:15:19.219014 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:19.218990    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a84e4890-ba61-49a5-b9b6-be9cf741d2d8-workload-socket" pod "a84e4890-ba61-49a5-b9b6-be9cf741d2d8" (UID: "a84e4890-ba61-49a5-b9b6-be9cf741d2d8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:15:19.219064 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:19.219031    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a84e4890-ba61-49a5-b9b6-be9cf741d2d8-istiod-ca-cert" pod "a84e4890-ba61-49a5-b9b6-be9cf741d2d8" (UID: "a84e4890-ba61-49a5-b9b6-be9cf741d2d8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:15:19.219378 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:19.219351    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a84e4890-ba61-49a5-b9b6-be9cf741d2d8-workload-certs" pod "a84e4890-ba61-49a5-b9b6-be9cf741d2d8" (UID: "a84e4890-ba61-49a5-b9b6-be9cf741d2d8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:15:19.219378 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:19.219374    5061 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a84e4890-ba61-49a5-b9b6-be9cf741d2d8-credential-socket\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:19.219539 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:19.219393    5061 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a84e4890-ba61-49a5-b9b6-be9cf741d2d8-istio-data\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:19.219539 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:19.219407    5061 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a84e4890-ba61-49a5-b9b6-be9cf741d2d8-istiod-ca-cert\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:19.219539 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:19.219420    5061 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a84e4890-ba61-49a5-b9b6-be9cf741d2d8-workload-socket\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:19.220768 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:19.220742    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a84e4890-ba61-49a5-b9b6-be9cf741d2d8-istio-podinfo" pod "a84e4890-ba61-49a5-b9b6-be9cf741d2d8" (UID: "a84e4890-ba61-49a5-b9b6-be9cf741d2d8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:15:19.221358 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:19.221333    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a84e4890-ba61-49a5-b9b6-be9cf741d2d8-kube-api-access-mfm9k" pod "a84e4890-ba61-49a5-b9b6-be9cf741d2d8" (UID: "a84e4890-ba61-49a5-b9b6-be9cf741d2d8"). InnerVolumeSpecName "kube-api-access-mfm9k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:19.221663 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:19.221636    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a84e4890-ba61-49a5-b9b6-be9cf741d2d8-istio-envoy" pod "a84e4890-ba61-49a5-b9b6-be9cf741d2d8" (UID: "a84e4890-ba61-49a5-b9b6-be9cf741d2d8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:15:19.222479 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:19.222459    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a84e4890-ba61-49a5-b9b6-be9cf741d2d8-istio-token" pod "a84e4890-ba61-49a5-b9b6-be9cf741d2d8" (UID: "a84e4890-ba61-49a5-b9b6-be9cf741d2d8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:19.222833 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:19.222810    5061 generic.go:356] "Generic (PLEG): container finished" podID="799953d1-1dc8-421e-9e98-d2696d399ac2" containerID="8eddbddec2bf4fa136c7343e98b86ac5eb26c4de459abf34ac50a87be0e862a6" exitCode=0
Oct 30 22:15:19.222930 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:19.222874    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-n4bnf"
Oct 30 22:15:19.222930 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:19.222892    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-n4bnf" event={"ID":"799953d1-1dc8-421e-9e98-d2696d399ac2","Type":"ContainerDied","Data":"8eddbddec2bf4fa136c7343e98b86ac5eb26c4de459abf34ac50a87be0e862a6"}
Oct 30 22:15:19.223038 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:19.222934    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-n4bnf" event={"ID":"799953d1-1dc8-421e-9e98-d2696d399ac2","Type":"ContainerDied","Data":"7995f901a3e0e9925c44d43afa72388bb6d450c87cd051cfaf794141c9cfae04"}
Oct 30 22:15:19.223241 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:19.223223    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-0ce563-openshift-default-f75cc7fc9-445r7"
Oct 30 22:15:19.224239 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:19.224219    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-0ce563-openshift-default-f75cc7fc9-445r7"
Oct 30 22:15:19.225601 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:19.225586    5061 scope.go:122] "RemoveContainer" containerID="07fdf70a88a04ac7efc92c5e49d0ebd555edfb6f1566d39343b15b8293059b8c"
Oct 30 22:15:19.234896 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:19.234832    5061 scope.go:122] "RemoveContainer" containerID="4b9f57dc4c1733ce47e979b2d62225e7a7913ac075487123a961cfd2366c525f"
Oct 30 22:15:19.243801 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:19.243785    5061 scope.go:122] "RemoveContainer" containerID="82dddbddcfcce85bc3d85b3bf22ef211a56c6bc33ad80549fee2923e0cdf341b"
Oct 30 22:15:19.256944 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:19.256927    5061 scope.go:122] "RemoveContainer" containerID="31464249c4951adb2f8ba7b6f6eec05cc32771225c8cc082b82dee5caeec4f25"
Oct 30 22:15:19.270566 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:19.270517    5061 scope.go:122] "RemoveContainer" containerID="8eddbddec2bf4fa136c7343e98b86ac5eb26c4de459abf34ac50a87be0e862a6"
Oct 30 22:15:19.296730 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:19.290661    5061 scope.go:122] "RemoveContainer" containerID="8eddbddec2bf4fa136c7343e98b86ac5eb26c4de459abf34ac50a87be0e862a6"
Oct 30 22:15:19.299510 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:15:19.296934    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8eddbddec2bf4fa136c7343e98b86ac5eb26c4de459abf34ac50a87be0e862a6\": container with ID starting with 8eddbddec2bf4fa136c7343e98b86ac5eb26c4de459abf34ac50a87be0e862a6 not found: ID does not exist" containerID="8eddbddec2bf4fa136c7343e98b86ac5eb26c4de459abf34ac50a87be0e862a6"
Oct 30 22:15:19.299510 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:19.296975    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8eddbddec2bf4fa136c7343e98b86ac5eb26c4de459abf34ac50a87be0e862a6"} err="failed to get container status \"8eddbddec2bf4fa136c7343e98b86ac5eb26c4de459abf34ac50a87be0e862a6\": rpc error: code = NotFound desc = could not find container \"8eddbddec2bf4fa136c7343e98b86ac5eb26c4de459abf34ac50a87be0e862a6\": container with ID starting with 8eddbddec2bf4fa136c7343e98b86ac5eb26c4de459abf34ac50a87be0e862a6 not found: ID does not exist"
Oct 30 22:15:19.306639 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:19.306612    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-4d8041-29889975-fn5gn"]
Oct 30 22:15:19.320992 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:19.320964    5061 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a84e4890-ba61-49a5-b9b6-be9cf741d2d8-workload-certs\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:19.320992 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:19.320991    5061 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a84e4890-ba61-49a5-b9b6-be9cf741d2d8-istio-podinfo\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:19.321147 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:19.321003    5061 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a84e4890-ba61-49a5-b9b6-be9cf741d2d8-istio-token\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:19.321147 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:19.321012    5061 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a84e4890-ba61-49a5-b9b6-be9cf741d2d8-istio-envoy\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:19.321147 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:19.321021    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfm9k\" (UniqueName: \"kubernetes.io/projected/a84e4890-ba61-49a5-b9b6-be9cf741d2d8-kube-api-access-mfm9k\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:19.397523 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:19.397480    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-778d39d3-29889975-mct28"]
Oct 30 22:15:19.467929 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:19.467900    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-4d8041-29889975-fn5gn"
Oct 30 22:15:19.532600 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:19.532566    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42adc96b-1b21-484e-b6fa-51bbf1127ac0-kube-api-access-gf2fw\" (UniqueName: \"kubernetes.io/projected/42adc96b-1b21-484e-b6fa-51bbf1127ac0-kube-api-access-gf2fw\") pod \"42adc96b-1b21-484e-b6fa-51bbf1127ac0\" (UID: \"42adc96b-1b21-484e-b6fa-51bbf1127ac0\") "
Oct 30 22:15:19.532831 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:19.532807    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42adc96b-1b21-484e-b6fa-51bbf1127ac0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42adc96b-1b21-484e-b6fa-51bbf1127ac0-maas-api-service-ca\") pod \"42adc96b-1b21-484e-b6fa-51bbf1127ac0\" (UID: \"42adc96b-1b21-484e-b6fa-51bbf1127ac0\") "
Oct 30 22:15:19.534443 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:19.534397    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42adc96b-1b21-484e-b6fa-51bbf1127ac0-maas-api-service-ca" pod "42adc96b-1b21-484e-b6fa-51bbf1127ac0" (UID: "42adc96b-1b21-484e-b6fa-51bbf1127ac0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:15:19.536718 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:19.536691    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42adc96b-1b21-484e-b6fa-51bbf1127ac0-kube-api-access-gf2fw" pod "42adc96b-1b21-484e-b6fa-51bbf1127ac0" (UID: "42adc96b-1b21-484e-b6fa-51bbf1127ac0"). InnerVolumeSpecName "kube-api-access-gf2fw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:19.605387 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:19.605365    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-778d39d3-29889975-mct28"
Oct 30 22:15:19.636122 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:19.634595    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3787aeb2-4e6e-4855-8be8-137ebcd39a26-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3787aeb2-4e6e-4855-8be8-137ebcd39a26-maas-api-service-ca\") pod \"3787aeb2-4e6e-4855-8be8-137ebcd39a26\" (UID: \"3787aeb2-4e6e-4855-8be8-137ebcd39a26\") "
Oct 30 22:15:19.636122 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:19.634991    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3787aeb2-4e6e-4855-8be8-137ebcd39a26-maas-api-service-ca" pod "3787aeb2-4e6e-4855-8be8-137ebcd39a26" (UID: "3787aeb2-4e6e-4855-8be8-137ebcd39a26"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:15:19.636122 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:19.635210    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42adc96b-1b21-484e-b6fa-51bbf1127ac0-maas-api-service-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:19.636122 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:19.635225    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3787aeb2-4e6e-4855-8be8-137ebcd39a26-maas-api-service-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:19.636122 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:19.635234    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gf2fw\" (UniqueName: \"kubernetes.io/projected/42adc96b-1b21-484e-b6fa-51bbf1127ac0-kube-api-access-gf2fw\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:19.735720 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:19.735631    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5402dd-29889975-jjcmb"]
Oct 30 22:15:19.736596 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:19.736397    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3787aeb2-4e6e-4855-8be8-137ebcd39a26-kube-api-access-llp8p\" (UniqueName: \"kubernetes.io/projected/3787aeb2-4e6e-4855-8be8-137ebcd39a26-kube-api-access-llp8p\") pod \"3787aeb2-4e6e-4855-8be8-137ebcd39a26\" (UID: \"3787aeb2-4e6e-4855-8be8-137ebcd39a26\") "
Oct 30 22:15:19.738594 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:19.738568    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3787aeb2-4e6e-4855-8be8-137ebcd39a26-kube-api-access-llp8p" pod "3787aeb2-4e6e-4855-8be8-137ebcd39a26" (UID: "3787aeb2-4e6e-4855-8be8-137ebcd39a26"). InnerVolumeSpecName "kube-api-access-llp8p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:19.839263 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:19.839232    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-llp8p\" (UniqueName: \"kubernetes.io/projected/3787aeb2-4e6e-4855-8be8-137ebcd39a26-kube-api-access-llp8p\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:19.964882 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:19.960218    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-02bd3b-29889975-jjfj2"]
Oct 30 22:15:19.987419 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:19.987055    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5402dd-29889975-jjcmb"
Oct 30 22:15:20.043678 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:20.043635    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ba41c1f-1fc3-4a32-967c-2d86878e2887-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ba41c1f-1fc3-4a32-967c-2d86878e2887-maas-api-service-ca\") pod \"0ba41c1f-1fc3-4a32-967c-2d86878e2887\" (UID: \"0ba41c1f-1fc3-4a32-967c-2d86878e2887\") "
Oct 30 22:15:20.045931 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:20.043730    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ba41c1f-1fc3-4a32-967c-2d86878e2887-kube-api-access-mcbck\" (UniqueName: \"kubernetes.io/projected/0ba41c1f-1fc3-4a32-967c-2d86878e2887-kube-api-access-mcbck\") pod \"0ba41c1f-1fc3-4a32-967c-2d86878e2887\" (UID: \"0ba41c1f-1fc3-4a32-967c-2d86878e2887\") "
Oct 30 22:15:20.045931 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:20.045912    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ba41c1f-1fc3-4a32-967c-2d86878e2887-maas-api-service-ca" pod "0ba41c1f-1fc3-4a32-967c-2d86878e2887" (UID: "0ba41c1f-1fc3-4a32-967c-2d86878e2887"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:15:20.048786 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:20.048370    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ba41c1f-1fc3-4a32-967c-2d86878e2887-kube-api-access-mcbck" pod "0ba41c1f-1fc3-4a32-967c-2d86878e2887" (UID: "0ba41c1f-1fc3-4a32-967c-2d86878e2887"). InnerVolumeSpecName "kube-api-access-mcbck". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:20.070800 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:20.070698    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-25de17-29889975-qk5j5"]
Oct 30 22:15:20.104334 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:20.104281    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e66edb-29889975-flzf6"]
Oct 30 22:15:20.111026 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:20.110752    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e66edb-29889975-flzf6"]
Oct 30 22:15:20.129512 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:20.129487    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-02bd3b-29889975-jjfj2"
Oct 30 22:15:20.145009 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:20.144978    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06e58c0b-aa5c-4f1e-8228-7070a03657bb-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06e58c0b-aa5c-4f1e-8228-7070a03657bb-maas-api-service-ca\") pod \"06e58c0b-aa5c-4f1e-8228-7070a03657bb\" (UID: \"06e58c0b-aa5c-4f1e-8228-7070a03657bb\") "
Oct 30 22:15:20.145172 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:20.145058    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06e58c0b-aa5c-4f1e-8228-7070a03657bb-kube-api-access-nb7jr\" (UniqueName: \"kubernetes.io/projected/06e58c0b-aa5c-4f1e-8228-7070a03657bb-kube-api-access-nb7jr\") pod \"06e58c0b-aa5c-4f1e-8228-7070a03657bb\" (UID: \"06e58c0b-aa5c-4f1e-8228-7070a03657bb\") "
Oct 30 22:15:20.145352 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:20.145328    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06e58c0b-aa5c-4f1e-8228-7070a03657bb-maas-api-service-ca" pod "06e58c0b-aa5c-4f1e-8228-7070a03657bb" (UID: "06e58c0b-aa5c-4f1e-8228-7070a03657bb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:15:20.145948 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:20.145928    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06e58c0b-aa5c-4f1e-8228-7070a03657bb-maas-api-service-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:20.146048 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:20.145951    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ba41c1f-1fc3-4a32-967c-2d86878e2887-maas-api-service-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:20.146048 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:20.145965    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcbck\" (UniqueName: \"kubernetes.io/projected/0ba41c1f-1fc3-4a32-967c-2d86878e2887-kube-api-access-mcbck\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:20.147054 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:20.147035    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06e58c0b-aa5c-4f1e-8228-7070a03657bb-kube-api-access-nb7jr" pod "06e58c0b-aa5c-4f1e-8228-7070a03657bb" (UID: "06e58c0b-aa5c-4f1e-8228-7070a03657bb"). InnerVolumeSpecName "kube-api-access-nb7jr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:20.180896 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:20.180789    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-6ea8da-29889975-r8jsf"]
Oct 30 22:15:20.210485 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:20.209964    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-778d39d3-d4gvh" podStartSLOduration=3.209946337 podStartE2EDuration="3.209946337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:20.205705472 +0000 UTC m=+1973.818475231" watchObservedRunningTime="2026-10-30 22:15:20.209946337 +0000 UTC m=+1973.822716102"
Oct 30 22:15:20.236249 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:20.236221    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-4d8041-29889975-fn5gn"
Oct 30 22:15:20.236249 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:20.236235    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-4d8041-29889975-fn5gn" event={"ID":"42adc96b-1b21-484e-b6fa-51bbf1127ac0","Type":"ContainerDied","Data":"75e1e5ca3c4f2da04268187cf485f66fce8f323743e6227061689bc2cefa18ed"}
Oct 30 22:15:20.236468 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:20.236301    5061 scope.go:122] "RemoveContainer" containerID="eaeae843ba7656c9a28037048ab31364e8efec6a884f7468f2ddd1c62e3136f1"
Oct 30 22:15:20.240184 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:20.240090    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-778d39d3-29889975-mct28"
Oct 30 22:15:20.240184 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:20.240108    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-778d39d3-29889975-mct28" event={"ID":"3787aeb2-4e6e-4855-8be8-137ebcd39a26","Type":"ContainerDied","Data":"753bd4d64d3f6d0357ee517d614c3787d67381e5ba79679cc9f476c2c3c5ad6b"}
Oct 30 22:15:20.242925 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:20.242898    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-02bd3b-29889975-jjfj2"
Oct 30 22:15:20.243029 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:20.242921    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-02bd3b-29889975-jjfj2" event={"ID":"06e58c0b-aa5c-4f1e-8228-7070a03657bb","Type":"ContainerDied","Data":"72966f1dbddeb6aa1d9643bd82e3aef651da682cb5b9233644ee3e1579518fe9"}
Oct 30 22:15:20.244782 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:20.244755    5061 generic.go:356] "Generic (PLEG): container finished" podID="2e7528ce-43ab-4e4c-be3f-b5a87b82d3b5" containerID="9db031599672d729bea66c018b477c94a91be9a60623547868bb62fc1213d37a" exitCode=0
Oct 30 22:15:20.244898 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:20.244843    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-778d39d3-d4gvh" event={"ID":"2e7528ce-43ab-4e4c-be3f-b5a87b82d3b5","Type":"ContainerDied","Data":"9db031599672d729bea66c018b477c94a91be9a60623547868bb62fc1213d37a"}
Oct 30 22:15:20.247420 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:20.247396    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nb7jr\" (UniqueName: \"kubernetes.io/projected/06e58c0b-aa5c-4f1e-8228-7070a03657bb-kube-api-access-nb7jr\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:20.251507 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:20.251453    5061 scope.go:122] "RemoveContainer" containerID="0b2dde3bc4be057d8f2e457b1fe85cbf4cb48e6903c66b54dfc322db1064ee02"
Oct 30 22:15:20.253580 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:20.253553    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5402dd-29889975-jjcmb" event={"ID":"0ba41c1f-1fc3-4a32-967c-2d86878e2887","Type":"ContainerDied","Data":"a4f7142dc6a02edfb7a68046d38c0650e316c7a877c46d032784c3f5810f1c61"}
Oct 30 22:15:20.253580 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:20.253573    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5402dd-29889975-jjcmb"
Oct 30 22:15:20.265715 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:20.265693    5061 scope.go:122] "RemoveContainer" containerID="c6360753e6f88eb6cd0d0b3b68a9d7b3a6229d379fb03b86e5813ca18544f445"
Oct 30 22:15:20.269343 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:20.269320    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-25de17-29889975-qk5j5"
Oct 30 22:15:20.276000 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:20.275984    5061 scope.go:122] "RemoveContainer" containerID="a31b3bea1a44d9a45d4e8fe0aa6c1588b8f5dc76a564fc9cb90ef66e2d1c3770"
Oct 30 22:15:20.315873 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:20.315842    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-5z645"]
Oct 30 22:15:20.320279 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:20.320255    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-5z645"]
Oct 30 22:15:20.348711 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:20.348679    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/777f3189-c018-4cdb-92ba-eeb10b71e868-kube-api-access-j6cw8\" (UniqueName: \"kubernetes.io/projected/777f3189-c018-4cdb-92ba-eeb10b71e868-kube-api-access-j6cw8\") pod \"777f3189-c018-4cdb-92ba-eeb10b71e868\" (UID: \"777f3189-c018-4cdb-92ba-eeb10b71e868\") "
Oct 30 22:15:20.348861 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:20.348760    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/777f3189-c018-4cdb-92ba-eeb10b71e868-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/777f3189-c018-4cdb-92ba-eeb10b71e868-maas-api-service-ca\") pod \"777f3189-c018-4cdb-92ba-eeb10b71e868\" (UID: \"777f3189-c018-4cdb-92ba-eeb10b71e868\") "
Oct 30 22:15:20.349145 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:20.349119    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/777f3189-c018-4cdb-92ba-eeb10b71e868-maas-api-service-ca" pod "777f3189-c018-4cdb-92ba-eeb10b71e868" (UID: "777f3189-c018-4cdb-92ba-eeb10b71e868"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:15:20.350185 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:20.350160    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/777f3189-c018-4cdb-92ba-eeb10b71e868-maas-api-service-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:20.351408 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:20.351384    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/777f3189-c018-4cdb-92ba-eeb10b71e868-kube-api-access-j6cw8" pod "777f3189-c018-4cdb-92ba-eeb10b71e868" (UID: "777f3189-c018-4cdb-92ba-eeb10b71e868"). InnerVolumeSpecName "kube-api-access-j6cw8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:20.396787 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:20.396762    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-6ea8da-29889975-r8jsf"
Oct 30 22:15:20.451025 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:20.450989    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08aac4a2-7d84-4b74-bf47-dc849fd6e446-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08aac4a2-7d84-4b74-bf47-dc849fd6e446-maas-api-service-ca\") pod \"08aac4a2-7d84-4b74-bf47-dc849fd6e446\" (UID: \"08aac4a2-7d84-4b74-bf47-dc849fd6e446\") "
Oct 30 22:15:20.451300 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:20.451261    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08aac4a2-7d84-4b74-bf47-dc849fd6e446-kube-api-access-ngvv2\" (UniqueName: \"kubernetes.io/projected/08aac4a2-7d84-4b74-bf47-dc849fd6e446-kube-api-access-ngvv2\") pod \"08aac4a2-7d84-4b74-bf47-dc849fd6e446\" (UID: \"08aac4a2-7d84-4b74-bf47-dc849fd6e446\") "
Oct 30 22:15:20.451385 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:20.451327    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08aac4a2-7d84-4b74-bf47-dc849fd6e446-maas-api-service-ca" pod "08aac4a2-7d84-4b74-bf47-dc849fd6e446" (UID: "08aac4a2-7d84-4b74-bf47-dc849fd6e446"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:15:20.451803 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:20.451784    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6cw8\" (UniqueName: \"kubernetes.io/projected/777f3189-c018-4cdb-92ba-eeb10b71e868-kube-api-access-j6cw8\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:20.451883 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:20.451805    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08aac4a2-7d84-4b74-bf47-dc849fd6e446-maas-api-service-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:20.453356 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:20.453336    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08aac4a2-7d84-4b74-bf47-dc849fd6e446-kube-api-access-ngvv2" pod "08aac4a2-7d84-4b74-bf47-dc849fd6e446" (UID: "08aac4a2-7d84-4b74-bf47-dc849fd6e446"). InnerVolumeSpecName "kube-api-access-ngvv2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:20.537183 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:20.537092    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b4f58e4-2611-45f1-b431-a33ce392d964" path="/var/lib/kubelet/pods/8b4f58e4-2611-45f1-b431-a33ce392d964/volumes"
Oct 30 22:15:20.538053 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:20.538024    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a84e4890-ba61-49a5-b9b6-be9cf741d2d8" path="/var/lib/kubelet/pods/a84e4890-ba61-49a5-b9b6-be9cf741d2d8/volumes"
Oct 30 22:15:20.552962 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:20.552929    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngvv2\" (UniqueName: \"kubernetes.io/projected/08aac4a2-7d84-4b74-bf47-dc849fd6e446-kube-api-access-ngvv2\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:20.883660 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:20.883624    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-n4bnf"]
Oct 30 22:15:20.887686 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:20.887655    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-n4bnf"]
Oct 30 22:15:21.100018 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:21.099983    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889975-dq9xr"]
Oct 30 22:15:21.103891 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:21.103834    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889975-dq9xr"]
Oct 30 22:15:21.206952 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:21.206853    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-778d39d3-29889975-mct28"]
Oct 30 22:15:21.212086 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:21.212056    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-778d39d3-29889975-mct28"]
Oct 30 22:15:21.271562 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:21.271501    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-6ea8da-29889975-r8jsf"
Oct 30 22:15:21.271562 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:21.271509    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-6ea8da-29889975-r8jsf" event={"ID":"08aac4a2-7d84-4b74-bf47-dc849fd6e446","Type":"ContainerDied","Data":"ef79113642d3e216ccfa7fe792ee52b8a153464d312307fea798ce4c7d0d0888"}
Oct 30 22:15:21.271769 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:21.271592    5061 scope.go:122] "RemoveContainer" containerID="a811babd464d25f7342e6f44d615cec374723f76293f8e44ec1d72a8794e45f9"
Oct 30 22:15:21.273081 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:21.273058    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-25de17-29889975-qk5j5" event={"ID":"777f3189-c018-4cdb-92ba-eeb10b71e868","Type":"ContainerDied","Data":"69df25cba20b08dd834f32fd4448cf84312170dcad2ddee4dea97fed34ef3ad5"}
Oct 30 22:15:21.273585 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:21.273071    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-25de17-29889975-qk5j5"
Oct 30 22:15:21.283192 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:21.283170    5061 scope.go:122] "RemoveContainer" containerID="2009ea126331a2f9681f403a8cb2cffde983feed727bc307db936b4b3d605b7e"
Oct 30 22:15:21.377512 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:21.377472    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5402dd-29889975-jjcmb"]
Oct 30 22:15:21.381590 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:21.381563    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5402dd-29889975-jjcmb"]
Oct 30 22:15:21.435583 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:21.435541    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5fef81-29889975-fdwmx"]
Oct 30 22:15:21.484227 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:21.484160    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-778d39d3-d4gvh"
Oct 30 22:15:21.567089 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:21.567042    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-02bd3b-29889975-jjfj2"]
Oct 30 22:15:21.571057 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:21.571026    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e7528ce-43ab-4e4c-be3f-b5a87b82d3b5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e7528ce-43ab-4e4c-be3f-b5a87b82d3b5-maas-api-xks-ca\") pod \"2e7528ce-43ab-4e4c-be3f-b5a87b82d3b5\" (UID: \"2e7528ce-43ab-4e4c-be3f-b5a87b82d3b5\") "
Oct 30 22:15:21.571198 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:21.571153    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e7528ce-43ab-4e4c-be3f-b5a87b82d3b5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e7528ce-43ab-4e4c-be3f-b5a87b82d3b5-maas-api-openshift-ca\") pod \"2e7528ce-43ab-4e4c-be3f-b5a87b82d3b5\" (UID: \"2e7528ce-43ab-4e4c-be3f-b5a87b82d3b5\") "
Oct 30 22:15:21.571277 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:21.571258    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e7528ce-43ab-4e4c-be3f-b5a87b82d3b5-kube-api-access-9bxh5\" (UniqueName: \"kubernetes.io/projected/2e7528ce-43ab-4e4c-be3f-b5a87b82d3b5-kube-api-access-9bxh5\") pod \"2e7528ce-43ab-4e4c-be3f-b5a87b82d3b5\" (UID: \"2e7528ce-43ab-4e4c-be3f-b5a87b82d3b5\") "
Oct 30 22:15:21.572135 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:21.572102    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e7528ce-43ab-4e4c-be3f-b5a87b82d3b5-maas-api-openshift-ca" pod "2e7528ce-43ab-4e4c-be3f-b5a87b82d3b5" (UID: "2e7528ce-43ab-4e4c-be3f-b5a87b82d3b5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:15:21.575590 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:21.573357    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-02bd3b-29889975-jjfj2"]
Oct 30 22:15:21.577849 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:21.577822    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e7528ce-43ab-4e4c-be3f-b5a87b82d3b5-maas-api-xks-ca" pod "2e7528ce-43ab-4e4c-be3f-b5a87b82d3b5" (UID: "2e7528ce-43ab-4e4c-be3f-b5a87b82d3b5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:15:21.578475 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:21.577845    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e7528ce-43ab-4e4c-be3f-b5a87b82d3b5-kube-api-access-9bxh5" pod "2e7528ce-43ab-4e4c-be3f-b5a87b82d3b5" (UID: "2e7528ce-43ab-4e4c-be3f-b5a87b82d3b5"). InnerVolumeSpecName "kube-api-access-9bxh5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:21.624675 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:21.624651    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5fef81-29889975-fdwmx"
Oct 30 22:15:21.672237 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:21.672206    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-411ff1-29889975-fc8l4"]
Oct 30 22:15:21.672690 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:21.672650    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbaff831-8881-4762-8332-25c8eb4c2a28-kube-api-access-qsb6m\" (UniqueName: \"kubernetes.io/projected/cbaff831-8881-4762-8332-25c8eb4c2a28-kube-api-access-qsb6m\") pod \"cbaff831-8881-4762-8332-25c8eb4c2a28\" (UID: \"cbaff831-8881-4762-8332-25c8eb4c2a28\") "
Oct 30 22:15:21.672971 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:21.672816    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbaff831-8881-4762-8332-25c8eb4c2a28-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbaff831-8881-4762-8332-25c8eb4c2a28-maas-api-service-ca\") pod \"cbaff831-8881-4762-8332-25c8eb4c2a28\" (UID: \"cbaff831-8881-4762-8332-25c8eb4c2a28\") "
Oct 30 22:15:21.673351 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:21.673332    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e7528ce-43ab-4e4c-be3f-b5a87b82d3b5-maas-api-xks-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:21.673431 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:21.673357    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e7528ce-43ab-4e4c-be3f-b5a87b82d3b5-maas-api-openshift-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:21.673431 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:21.673372    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bxh5\" (UniqueName: \"kubernetes.io/projected/2e7528ce-43ab-4e4c-be3f-b5a87b82d3b5-kube-api-access-9bxh5\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:21.673431 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:21.673399    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbaff831-8881-4762-8332-25c8eb4c2a28-maas-api-service-ca" pod "cbaff831-8881-4762-8332-25c8eb4c2a28" (UID: "cbaff831-8881-4762-8332-25c8eb4c2a28"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:15:21.675177 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:21.675152    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbaff831-8881-4762-8332-25c8eb4c2a28-kube-api-access-qsb6m" pod "cbaff831-8881-4762-8332-25c8eb4c2a28" (UID: "cbaff831-8881-4762-8332-25c8eb4c2a28"). InnerVolumeSpecName "kube-api-access-qsb6m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:21.774865 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:21.774834    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qsb6m\" (UniqueName: \"kubernetes.io/projected/cbaff831-8881-4762-8332-25c8eb4c2a28-kube-api-access-qsb6m\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:21.774865 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:21.774862    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbaff831-8881-4762-8332-25c8eb4c2a28-maas-api-service-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:21.866893 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:21.866867    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-411ff1-29889975-fc8l4"
Oct 30 22:15:21.978485 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:21.977741    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52312d9d-e2f8-4ad3-bce1-53ffc5b08d4f-kube-api-access-lfkks\" (UniqueName: \"kubernetes.io/projected/52312d9d-e2f8-4ad3-bce1-53ffc5b08d4f-kube-api-access-lfkks\") pod \"52312d9d-e2f8-4ad3-bce1-53ffc5b08d4f\" (UID: \"52312d9d-e2f8-4ad3-bce1-53ffc5b08d4f\") "
Oct 30 22:15:21.978485 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:21.977896    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52312d9d-e2f8-4ad3-bce1-53ffc5b08d4f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52312d9d-e2f8-4ad3-bce1-53ffc5b08d4f-maas-api-service-ca\") pod \"52312d9d-e2f8-4ad3-bce1-53ffc5b08d4f\" (UID: \"52312d9d-e2f8-4ad3-bce1-53ffc5b08d4f\") "
Oct 30 22:15:21.978485 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:21.978471    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52312d9d-e2f8-4ad3-bce1-53ffc5b08d4f-maas-api-service-ca" pod "52312d9d-e2f8-4ad3-bce1-53ffc5b08d4f" (UID: "52312d9d-e2f8-4ad3-bce1-53ffc5b08d4f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:15:21.980353 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:21.980325    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52312d9d-e2f8-4ad3-bce1-53ffc5b08d4f-kube-api-access-lfkks" pod "52312d9d-e2f8-4ad3-bce1-53ffc5b08d4f" (UID: "52312d9d-e2f8-4ad3-bce1-53ffc5b08d4f"). InnerVolumeSpecName "kube-api-access-lfkks". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:22.076143 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:22.076109    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-4d8041-29889975-fn5gn"]
Oct 30 22:15:22.079679 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:22.079652    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52312d9d-e2f8-4ad3-bce1-53ffc5b08d4f-maas-api-service-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:22.079679 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:22.079677    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfkks\" (UniqueName: \"kubernetes.io/projected/52312d9d-e2f8-4ad3-bce1-53ffc5b08d4f-kube-api-access-lfkks\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:22.088572 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:22.088544    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-4d8041-29889975-fn5gn"]
Oct 30 22:15:22.283617 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:22.283586    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-411ff1-29889975-fc8l4"
Oct 30 22:15:22.283771 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:22.283640    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-411ff1-29889975-fc8l4" event={"ID":"52312d9d-e2f8-4ad3-bce1-53ffc5b08d4f","Type":"ContainerDied","Data":"030ec511c379cfeb5123d2e3a2fee91af15a7c9d7bd6ac327f186c6114c8f84b"}
Oct 30 22:15:22.283771 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:22.283697    5061 scope.go:122] "RemoveContainer" containerID="49e84ec08d7ae6bfe29d611f8bbe53eb6a64b6bfed0e377db6cd8b48a1555a04"
Oct 30 22:15:22.287468 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:22.287439    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5fef81-29889975-fdwmx"
Oct 30 22:15:22.287629 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:22.287485    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5fef81-29889975-fdwmx" event={"ID":"cbaff831-8881-4762-8332-25c8eb4c2a28","Type":"ContainerDied","Data":"22fa96e22af060d2623e1588a6083c0a2f5f39c2643f92231453fb16fe142549"}
Oct 30 22:15:22.289084 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:22.289060    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-778d39d3-d4gvh" event={"ID":"2e7528ce-43ab-4e4c-be3f-b5a87b82d3b5","Type":"ContainerDied","Data":"b9287ce9485abd742af383e54abb52ec5efb8a20d616156d00c5b31048e7ad41"}
Oct 30 22:15:22.289178 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:22.289098    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-778d39d3-d4gvh"
Oct 30 22:15:22.289178 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:22.289105    5061 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b9287ce9485abd742af383e54abb52ec5efb8a20d616156d00c5b31048e7ad41"
Oct 30 22:15:22.295415 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:22.295393    5061 scope.go:122] "RemoveContainer" containerID="cb32765cf6e669d9e8b83e6886e3f8e10b647297913e9ca34862d174f0b825e3"
Oct 30 22:15:22.329685 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:22.329634    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-6ea8da-29889975-r8jsf"]
Oct 30 22:15:22.339408 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:22.338918    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-6ea8da-29889975-r8jsf"]
Oct 30 22:15:22.529718 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:22.529681    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06e58c0b-aa5c-4f1e-8228-7070a03657bb" path="/var/lib/kubelet/pods/06e58c0b-aa5c-4f1e-8228-7070a03657bb/volumes"
Oct 30 22:15:22.530127 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:22.530113    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08aac4a2-7d84-4b74-bf47-dc849fd6e446" path="/var/lib/kubelet/pods/08aac4a2-7d84-4b74-bf47-dc849fd6e446/volumes"
Oct 30 22:15:22.530522 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:22.530509    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ba41c1f-1fc3-4a32-967c-2d86878e2887" path="/var/lib/kubelet/pods/0ba41c1f-1fc3-4a32-967c-2d86878e2887/volumes"
Oct 30 22:15:22.531288 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:22.531270    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3787aeb2-4e6e-4855-8be8-137ebcd39a26" path="/var/lib/kubelet/pods/3787aeb2-4e6e-4855-8be8-137ebcd39a26/volumes"
Oct 30 22:15:22.531726 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:22.531710    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42adc96b-1b21-484e-b6fa-51bbf1127ac0" path="/var/lib/kubelet/pods/42adc96b-1b21-484e-b6fa-51bbf1127ac0/volumes"
Oct 30 22:15:22.532172 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:22.532159    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="799953d1-1dc8-421e-9e98-d2696d399ac2" path="/var/lib/kubelet/pods/799953d1-1dc8-421e-9e98-d2696d399ac2/volumes"
Oct 30 22:15:22.532888 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:22.532873    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f220141f-4150-4f9c-9e14-631777d0ac14" path="/var/lib/kubelet/pods/f220141f-4150-4f9c-9e14-631777d0ac14/volumes"
Oct 30 22:15:22.673509 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:22.673475    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-25de17-29889975-qk5j5"]
Oct 30 22:15:22.677638 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:22.677611    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-25de17-29889975-qk5j5"]
Oct 30 22:15:22.782811 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:22.782779    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5fef81-29889975-fdwmx"]
Oct 30 22:15:22.786484 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:22.786453    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5fef81-29889975-fdwmx"]
Oct 30 22:15:22.891129 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:22.891049    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-411ff1-29889975-fc8l4"]
Oct 30 22:15:22.900685 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:22.900659    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-411ff1-29889975-fc8l4"]
Oct 30 22:15:24.529776 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:24.529735    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52312d9d-e2f8-4ad3-bce1-53ffc5b08d4f" path="/var/lib/kubelet/pods/52312d9d-e2f8-4ad3-bce1-53ffc5b08d4f/volumes"
Oct 30 22:15:24.530438 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:24.530416    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="777f3189-c018-4cdb-92ba-eeb10b71e868" path="/var/lib/kubelet/pods/777f3189-c018-4cdb-92ba-eeb10b71e868/volumes"
Oct 30 22:15:24.530946 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:24.530924    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbaff831-8881-4762-8332-25c8eb4c2a28" path="/var/lib/kubelet/pods/cbaff831-8881-4762-8332-25c8eb4c2a28/volumes"
Oct 30 22:15:24.669761 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:15:24.669718    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 22:15:25.106550 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:25.103129    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0ce563-6b8c6c95bc-fsrrg"]
Oct 30 22:15:25.110545 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:25.109576    5061 cpu_manager.go:414] "containerMap: removing container" podUID="42adc96b-1b21-484e-b6fa-51bbf1127ac0" containerName="cleanup"
Oct 30 22:15:25.110545 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:25.109622    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42adc96b-1b21-484e-b6fa-51bbf1127ac0" containerName="cleanup"
Oct 30 22:15:25.110545 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:25.109641    5061 cpu_manager.go:414] "containerMap: removing container" podUID="777f3189-c018-4cdb-92ba-eeb10b71e868" containerName="cleanup"
Oct 30 22:15:25.110545 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:25.109650    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="777f3189-c018-4cdb-92ba-eeb10b71e868" containerName="cleanup"
Oct 30 22:15:25.110545 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:25.109671    5061 cpu_manager.go:414] "containerMap: removing container" podUID="f220141f-4150-4f9c-9e14-631777d0ac14" containerName="cleanup"
Oct 30 22:15:25.110545 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:25.109680    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f220141f-4150-4f9c-9e14-631777d0ac14" containerName="cleanup"
Oct 30 22:15:25.110545 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:25.109715    5061 cpu_manager.go:414] "containerMap: removing container" podUID="f220141f-4150-4f9c-9e14-631777d0ac14" containerName="cleanup"
Oct 30 22:15:25.110545 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:25.109722    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f220141f-4150-4f9c-9e14-631777d0ac14" containerName="cleanup"
Oct 30 22:15:25.110545 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:25.109735    5061 cpu_manager.go:414] "containerMap: removing container" podUID="8b4f58e4-2611-45f1-b431-a33ce392d964" containerName="cleanup"
Oct 30 22:15:25.110545 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:25.109742    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b4f58e4-2611-45f1-b431-a33ce392d964" containerName="cleanup"
Oct 30 22:15:25.110545 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:25.109759    5061 cpu_manager.go:414] "containerMap: removing container" podUID="52312d9d-e2f8-4ad3-bce1-53ffc5b08d4f" containerName="cleanup"
Oct 30 22:15:25.110545 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:25.109767    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52312d9d-e2f8-4ad3-bce1-53ffc5b08d4f" containerName="cleanup"
Oct 30 22:15:25.110545 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:25.109782    5061 cpu_manager.go:414] "containerMap: removing container" podUID="06e58c0b-aa5c-4f1e-8228-7070a03657bb" containerName="cleanup"
Oct 30 22:15:25.110545 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:25.109790    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06e58c0b-aa5c-4f1e-8228-7070a03657bb" containerName="cleanup"
Oct 30 22:15:25.110545 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:25.109835    5061 cpu_manager.go:414] "containerMap: removing container" podUID="42adc96b-1b21-484e-b6fa-51bbf1127ac0" containerName="cleanup"
Oct 30 22:15:25.110545 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:25.109844    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42adc96b-1b21-484e-b6fa-51bbf1127ac0" containerName="cleanup"
Oct 30 22:15:25.110545 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:25.109867    5061 cpu_manager.go:414] "containerMap: removing container" podUID="799953d1-1dc8-421e-9e98-d2696d399ac2" containerName="istio-proxy"
Oct 30 22:15:25.110545 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:25.109875    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="799953d1-1dc8-421e-9e98-d2696d399ac2" containerName="istio-proxy"
Oct 30 22:15:25.110545 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:25.109886    5061 cpu_manager.go:414] "containerMap: removing container" podUID="0ba41c1f-1fc3-4a32-967c-2d86878e2887" containerName="cleanup"
Oct 30 22:15:25.110545 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:25.109893    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ba41c1f-1fc3-4a32-967c-2d86878e2887" containerName="cleanup"
Oct 30 22:15:25.110545 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:25.109949    5061 cpu_manager.go:414] "containerMap: removing container" podUID="a84e4890-ba61-49a5-b9b6-be9cf741d2d8" containerName="istio-proxy"
Oct 30 22:15:25.110545 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:25.109957    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a84e4890-ba61-49a5-b9b6-be9cf741d2d8" containerName="istio-proxy"
Oct 30 22:15:25.110545 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:25.109983    5061 cpu_manager.go:414] "containerMap: removing container" podUID="0ba41c1f-1fc3-4a32-967c-2d86878e2887" containerName="cleanup"
Oct 30 22:15:25.110545 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:25.109990    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ba41c1f-1fc3-4a32-967c-2d86878e2887" containerName="cleanup"
Oct 30 22:15:25.110545 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:25.110019    5061 cpu_manager.go:414] "containerMap: removing container" podUID="cbaff831-8881-4762-8332-25c8eb4c2a28" containerName="cleanup"
Oct 30 22:15:25.110545 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:25.110027    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbaff831-8881-4762-8332-25c8eb4c2a28" containerName="cleanup"
Oct 30 22:15:25.110545 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:25.110045    5061 cpu_manager.go:414] "containerMap: removing container" podUID="52312d9d-e2f8-4ad3-bce1-53ffc5b08d4f" containerName="cleanup"
Oct 30 22:15:25.110545 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:25.110052    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52312d9d-e2f8-4ad3-bce1-53ffc5b08d4f" containerName="cleanup"
Oct 30 22:15:25.110545 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:25.110070    5061 cpu_manager.go:414] "containerMap: removing container" podUID="06e58c0b-aa5c-4f1e-8228-7070a03657bb" containerName="cleanup"
Oct 30 22:15:25.110545 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:25.110079    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06e58c0b-aa5c-4f1e-8228-7070a03657bb" containerName="cleanup"
Oct 30 22:15:25.110545 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:25.110089    5061 cpu_manager.go:414] "containerMap: removing container" podUID="8b4f58e4-2611-45f1-b431-a33ce392d964" containerName="cleanup"
Oct 30 22:15:25.110545 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:25.110097    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b4f58e4-2611-45f1-b431-a33ce392d964" containerName="cleanup"
Oct 30 22:15:25.110545 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:25.110115    5061 cpu_manager.go:414] "containerMap: removing container" podUID="42adc96b-1b21-484e-b6fa-51bbf1127ac0" containerName="cleanup"
Oct 30 22:15:25.110545 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:25.110122    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42adc96b-1b21-484e-b6fa-51bbf1127ac0" containerName="cleanup"
Oct 30 22:15:25.110545 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:25.110141    5061 cpu_manager.go:414] "containerMap: removing container" podUID="3787aeb2-4e6e-4855-8be8-137ebcd39a26" containerName="cleanup"
Oct 30 22:15:25.110545 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:25.110149    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3787aeb2-4e6e-4855-8be8-137ebcd39a26" containerName="cleanup"
Oct 30 22:15:25.110545 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:25.110174    5061 cpu_manager.go:414] "containerMap: removing container" podUID="3787aeb2-4e6e-4855-8be8-137ebcd39a26" containerName="cleanup"
Oct 30 22:15:25.110545 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:25.110181    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3787aeb2-4e6e-4855-8be8-137ebcd39a26" containerName="cleanup"
Oct 30 22:15:25.110545 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:25.110191    5061 cpu_manager.go:414] "containerMap: removing container" podUID="0ba41c1f-1fc3-4a32-967c-2d86878e2887" containerName="cleanup"
Oct 30 22:15:25.110545 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:25.110199    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ba41c1f-1fc3-4a32-967c-2d86878e2887" containerName="cleanup"
Oct 30 22:15:25.110545 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:25.110216    5061 cpu_manager.go:414] "containerMap: removing container" podUID="cbaff831-8881-4762-8332-25c8eb4c2a28" containerName="cleanup"
Oct 30 22:15:25.110545 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:25.110223    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbaff831-8881-4762-8332-25c8eb4c2a28" containerName="cleanup"
Oct 30 22:15:25.110545 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:25.110241    5061 cpu_manager.go:414] "containerMap: removing container" podUID="2e7528ce-43ab-4e4c-be3f-b5a87b82d3b5" containerName="revoke-keys"
Oct 30 22:15:25.110545 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:25.110248    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e7528ce-43ab-4e4c-be3f-b5a87b82d3b5" containerName="revoke-keys"
Oct 30 22:15:25.110545 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:25.110270    5061 cpu_manager.go:414] "containerMap: removing container" podUID="8b4f58e4-2611-45f1-b431-a33ce392d964" containerName="cleanup"
Oct 30 22:15:25.110545 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:25.110278    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b4f58e4-2611-45f1-b431-a33ce392d964" containerName="cleanup"
Oct 30 22:15:25.110545 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:25.110293    5061 cpu_manager.go:414] "containerMap: removing container" podUID="777f3189-c018-4cdb-92ba-eeb10b71e868" containerName="cleanup"
Oct 30 22:15:25.110545 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:25.110300    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="777f3189-c018-4cdb-92ba-eeb10b71e868" containerName="cleanup"
Oct 30 22:15:25.110545 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:25.110311    5061 cpu_manager.go:414] "containerMap: removing container" podUID="06e58c0b-aa5c-4f1e-8228-7070a03657bb" containerName="cleanup"
Oct 30 22:15:25.110545 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:25.110328    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06e58c0b-aa5c-4f1e-8228-7070a03657bb" containerName="cleanup"
Oct 30 22:15:25.110545 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:25.110348    5061 cpu_manager.go:414] "containerMap: removing container" podUID="08aac4a2-7d84-4b74-bf47-dc849fd6e446" containerName="cleanup"
Oct 30 22:15:25.110545 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:25.110356    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08aac4a2-7d84-4b74-bf47-dc849fd6e446" containerName="cleanup"
Oct 30 22:15:25.110545 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:25.110370    5061 cpu_manager.go:414] "containerMap: removing container" podUID="08aac4a2-7d84-4b74-bf47-dc849fd6e446" containerName="cleanup"
Oct 30 22:15:25.110545 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:25.110378    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08aac4a2-7d84-4b74-bf47-dc849fd6e446" containerName="cleanup"
Oct 30 22:15:25.110545 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:25.110407    5061 cpu_manager.go:414] "containerMap: removing container" podUID="cbaff831-8881-4762-8332-25c8eb4c2a28" containerName="cleanup"
Oct 30 22:15:25.110545 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:25.110414    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbaff831-8881-4762-8332-25c8eb4c2a28" containerName="cleanup"
Oct 30 22:15:25.110545 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:25.110434    5061 cpu_manager.go:414] "containerMap: removing container" podUID="08aac4a2-7d84-4b74-bf47-dc849fd6e446" containerName="cleanup"
Oct 30 22:15:25.110545 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:25.110443    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08aac4a2-7d84-4b74-bf47-dc849fd6e446" containerName="cleanup"
Oct 30 22:15:25.113485 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:25.110929    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="42adc96b-1b21-484e-b6fa-51bbf1127ac0" containerName="cleanup"
Oct 30 22:15:25.113485 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:25.110950    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="799953d1-1dc8-421e-9e98-d2696d399ac2" containerName="istio-proxy"
Oct 30 22:15:25.113485 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:25.110970    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ba41c1f-1fc3-4a32-967c-2d86878e2887" containerName="cleanup"
Oct 30 22:15:25.113485 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:25.110985    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="f220141f-4150-4f9c-9e14-631777d0ac14" containerName="cleanup"
Oct 30 22:15:25.113485 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:25.111001    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="a84e4890-ba61-49a5-b9b6-be9cf741d2d8" containerName="istio-proxy"
Oct 30 22:15:25.113485 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:25.111020    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ba41c1f-1fc3-4a32-967c-2d86878e2887" containerName="cleanup"
Oct 30 22:15:25.113485 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:25.111039    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbaff831-8881-4762-8332-25c8eb4c2a28" containerName="cleanup"
Oct 30 22:15:25.113485 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:25.111047    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="52312d9d-e2f8-4ad3-bce1-53ffc5b08d4f" containerName="cleanup"
Oct 30 22:15:25.113485 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:25.111062    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="52312d9d-e2f8-4ad3-bce1-53ffc5b08d4f" containerName="cleanup"
Oct 30 22:15:25.113485 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:25.111078    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="06e58c0b-aa5c-4f1e-8228-7070a03657bb" containerName="cleanup"
Oct 30 22:15:25.113485 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:25.111086    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b4f58e4-2611-45f1-b431-a33ce392d964" containerName="cleanup"
Oct 30 22:15:25.113485 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:25.111095    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="42adc96b-1b21-484e-b6fa-51bbf1127ac0" containerName="cleanup"
Oct 30 22:15:25.113485 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:25.111112    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="3787aeb2-4e6e-4855-8be8-137ebcd39a26" containerName="cleanup"
Oct 30 22:15:25.113485 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:25.111130    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="3787aeb2-4e6e-4855-8be8-137ebcd39a26" containerName="cleanup"
Oct 30 22:15:25.113485 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:25.111145    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e7528ce-43ab-4e4c-be3f-b5a87b82d3b5" containerName="revoke-keys"
Oct 30 22:15:25.113485 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:25.111154    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b4f58e4-2611-45f1-b431-a33ce392d964" containerName="cleanup"
Oct 30 22:15:25.113485 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:25.111168    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="777f3189-c018-4cdb-92ba-eeb10b71e868" containerName="cleanup"
Oct 30 22:15:25.113485 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:25.111175    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="06e58c0b-aa5c-4f1e-8228-7070a03657bb" containerName="cleanup"
Oct 30 22:15:25.113485 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:25.111183    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="08aac4a2-7d84-4b74-bf47-dc849fd6e446" containerName="cleanup"
Oct 30 22:15:25.113485 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:25.111199    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="08aac4a2-7d84-4b74-bf47-dc849fd6e446" containerName="cleanup"
Oct 30 22:15:25.113485 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:25.111216    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbaff831-8881-4762-8332-25c8eb4c2a28" containerName="cleanup"
Oct 30 22:15:25.113485 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:25.111227    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="08aac4a2-7d84-4b74-bf47-dc849fd6e446" containerName="cleanup"
Oct 30 22:15:25.113485 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:25.111241    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="3787aeb2-4e6e-4855-8be8-137ebcd39a26" containerName="cleanup"
Oct 30 22:15:25.113485 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:25.111250    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="777f3189-c018-4cdb-92ba-eeb10b71e868" containerName="cleanup"
Oct 30 22:15:25.113485 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:25.111263    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="f220141f-4150-4f9c-9e14-631777d0ac14" containerName="cleanup"
Oct 30 22:15:25.113485 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:25.111275    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="f220141f-4150-4f9c-9e14-631777d0ac14" containerName="cleanup"
Oct 30 22:15:25.113485 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:25.111297    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b4f58e4-2611-45f1-b431-a33ce392d964" containerName="cleanup"
Oct 30 22:15:25.138036 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:25.127509    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0ce563-6b8c6c95bc-fsrrg"
Oct 30 22:15:25.155969 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:25.155246    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0ce563-6b8c6c95bc-fsrrg"]
Oct 30 22:15:25.195941 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:25.195910    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-0ce563\""
Oct 30 22:15:25.195941 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:25.195937    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-0ce563-dockercfg-8kh4f\""
Oct 30 22:15:25.230037 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:25.230001    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74b993cb-2ab2-4457-8885-a2ee906a351f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-0ce563-6b8c6c95bc-fsrrg\" (UID: \"74b993cb-2ab2-4457-8885-a2ee906a351f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0ce563-6b8c6c95bc-fsrrg"
Oct 30 22:15:25.230261 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:25.230215    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mhxx\" (UniqueName: \"kubernetes.io/projected/74b993cb-2ab2-4457-8885-a2ee906a351f-kube-api-access-6mhxx\") pod \"payload-pre-processing-e2e-api-b-0ce563-6b8c6c95bc-fsrrg\" (UID: \"74b993cb-2ab2-4457-8885-a2ee906a351f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0ce563-6b8c6c95bc-fsrrg"
Oct 30 22:15:25.256602 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:25.252136    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0ce563-f9b586484-rg56t"]
Oct 30 22:15:25.256602 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:25.253211    5061 cpu_manager.go:414] "containerMap: removing container" podUID="52312d9d-e2f8-4ad3-bce1-53ffc5b08d4f" containerName="cleanup"
Oct 30 22:15:25.256602 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:25.253232    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52312d9d-e2f8-4ad3-bce1-53ffc5b08d4f" containerName="cleanup"
Oct 30 22:15:25.256602 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:25.253255    5061 cpu_manager.go:414] "containerMap: removing container" podUID="3787aeb2-4e6e-4855-8be8-137ebcd39a26" containerName="cleanup"
Oct 30 22:15:25.256602 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:25.253263    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3787aeb2-4e6e-4855-8be8-137ebcd39a26" containerName="cleanup"
Oct 30 22:15:25.256602 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:25.253300    5061 cpu_manager.go:414] "containerMap: removing container" podUID="777f3189-c018-4cdb-92ba-eeb10b71e868" containerName="cleanup"
Oct 30 22:15:25.256602 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:25.253307    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="777f3189-c018-4cdb-92ba-eeb10b71e868" containerName="cleanup"
Oct 30 22:15:25.256602 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:25.253329    5061 cpu_manager.go:414] "containerMap: removing container" podUID="f220141f-4150-4f9c-9e14-631777d0ac14" containerName="cleanup"
Oct 30 22:15:25.256602 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:25.253337    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f220141f-4150-4f9c-9e14-631777d0ac14" containerName="cleanup"
Oct 30 22:15:25.256602 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:25.253549    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbaff831-8881-4762-8332-25c8eb4c2a28" containerName="cleanup"
Oct 30 22:15:25.256602 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:25.253563    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="777f3189-c018-4cdb-92ba-eeb10b71e868" containerName="cleanup"
Oct 30 22:15:25.256602 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:25.253584    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ba41c1f-1fc3-4a32-967c-2d86878e2887" containerName="cleanup"
Oct 30 22:15:25.256602 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:25.253597    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="52312d9d-e2f8-4ad3-bce1-53ffc5b08d4f" containerName="cleanup"
Oct 30 22:15:25.256602 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:25.253608    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="42adc96b-1b21-484e-b6fa-51bbf1127ac0" containerName="cleanup"
Oct 30 22:15:25.256602 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:25.253622    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="06e58c0b-aa5c-4f1e-8228-7070a03657bb" containerName="cleanup"
Oct 30 22:15:25.259149 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:25.259127    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-0ce563-f9b586484-rg56t"
Oct 30 22:15:25.322947 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:25.322900    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0ce563-f9b586484-rg56t"]
Oct 30 22:15:25.331686 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:25.331653    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92710489-bfc6-49db-b02d-4519d1b5f3c9-plugins-config-volume\") pod \"payload-processing-e2e-api-b-0ce563-f9b586484-rg56t\" (UID: \"92710489-bfc6-49db-b02d-4519d1b5f3c9\") " pod="openshift-ingress/payload-processing-e2e-api-b-0ce563-f9b586484-rg56t"
Oct 30 22:15:25.331968 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:25.331941    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74b993cb-2ab2-4457-8885-a2ee906a351f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-0ce563-6b8c6c95bc-fsrrg\" (UID: \"74b993cb-2ab2-4457-8885-a2ee906a351f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0ce563-6b8c6c95bc-fsrrg"
Oct 30 22:15:25.332162 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:25.332145    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ws74\" (UniqueName: \"kubernetes.io/projected/92710489-bfc6-49db-b02d-4519d1b5f3c9-kube-api-access-4ws74\") pod \"payload-processing-e2e-api-b-0ce563-f9b586484-rg56t\" (UID: \"92710489-bfc6-49db-b02d-4519d1b5f3c9\") " pod="openshift-ingress/payload-processing-e2e-api-b-0ce563-f9b586484-rg56t"
Oct 30 22:15:25.332337 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:25.332320    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mhxx\" (UniqueName: \"kubernetes.io/projected/74b993cb-2ab2-4457-8885-a2ee906a351f-kube-api-access-6mhxx\") pod \"payload-pre-processing-e2e-api-b-0ce563-6b8c6c95bc-fsrrg\" (UID: \"74b993cb-2ab2-4457-8885-a2ee906a351f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0ce563-6b8c6c95bc-fsrrg"
Oct 30 22:15:25.333686 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:25.333661    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74b993cb-2ab2-4457-8885-a2ee906a351f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-0ce563-6b8c6c95bc-fsrrg\" (UID: \"74b993cb-2ab2-4457-8885-a2ee906a351f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0ce563-6b8c6c95bc-fsrrg"
Oct 30 22:15:25.387275 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:25.387194    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mhxx\" (UniqueName: \"kubernetes.io/projected/74b993cb-2ab2-4457-8885-a2ee906a351f-kube-api-access-6mhxx\") pod \"payload-pre-processing-e2e-api-b-0ce563-6b8c6c95bc-fsrrg\" (UID: \"74b993cb-2ab2-4457-8885-a2ee906a351f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0ce563-6b8c6c95bc-fsrrg"
Oct 30 22:15:25.433597 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:25.433517    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ws74\" (UniqueName: \"kubernetes.io/projected/92710489-bfc6-49db-b02d-4519d1b5f3c9-kube-api-access-4ws74\") pod \"payload-processing-e2e-api-b-0ce563-f9b586484-rg56t\" (UID: \"92710489-bfc6-49db-b02d-4519d1b5f3c9\") " pod="openshift-ingress/payload-processing-e2e-api-b-0ce563-f9b586484-rg56t"
Oct 30 22:15:25.433761 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:25.433745    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92710489-bfc6-49db-b02d-4519d1b5f3c9-plugins-config-volume\") pod \"payload-processing-e2e-api-b-0ce563-f9b586484-rg56t\" (UID: \"92710489-bfc6-49db-b02d-4519d1b5f3c9\") " pod="openshift-ingress/payload-processing-e2e-api-b-0ce563-f9b586484-rg56t"
Oct 30 22:15:25.434429 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:25.434410    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92710489-bfc6-49db-b02d-4519d1b5f3c9-plugins-config-volume\") pod \"payload-processing-e2e-api-b-0ce563-f9b586484-rg56t\" (UID: \"92710489-bfc6-49db-b02d-4519d1b5f3c9\") " pod="openshift-ingress/payload-processing-e2e-api-b-0ce563-f9b586484-rg56t"
Oct 30 22:15:25.461614 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:25.461586    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0ce563-6b8c6c95bc-fsrrg"
Oct 30 22:15:25.487799 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:25.487766    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ws74\" (UniqueName: \"kubernetes.io/projected/92710489-bfc6-49db-b02d-4519d1b5f3c9-kube-api-access-4ws74\") pod \"payload-processing-e2e-api-b-0ce563-f9b586484-rg56t\" (UID: \"92710489-bfc6-49db-b02d-4519d1b5f3c9\") " pod="openshift-ingress/payload-processing-e2e-api-b-0ce563-f9b586484-rg56t"
Oct 30 22:15:25.630912 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:25.630876    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-0ce563-f9b586484-rg56t"
Oct 30 22:15:25.832499 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:25.832447    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0ce563-6b8c6c95bc-fsrrg"]
Oct 30 22:15:26.007389 ip-10-0-130-60 kubenswrapper[5061]: W1030 22:15:26.007361    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92710489_bfc6_49db_b02d_4519d1b5f3c9.slice/crio-1c8a01da12ce0ac09e0b982903e4ad718593a026ba4a3eda09e64def9a9268cb WatchSource:0}: Error finding container 1c8a01da12ce0ac09e0b982903e4ad718593a026ba4a3eda09e64def9a9268cb: Status 404 returned error can't find the container with id 1c8a01da12ce0ac09e0b982903e4ad718593a026ba4a3eda09e64def9a9268cb
Oct 30 22:15:26.008406 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:26.008375    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0ce563-f9b586484-rg56t"]
Oct 30 22:15:26.345907 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:26.345861    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-0ce563-f9b586484-rg56t" event={"ID":"92710489-bfc6-49db-b02d-4519d1b5f3c9","Type":"ContainerStarted","Data":"b1c8373c5f3cea31ee798be283c344de548e284ddf8973316cd5184ac2443624"}
Oct 30 22:15:26.346079 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:26.345932    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-0ce563-f9b586484-rg56t" event={"ID":"92710489-bfc6-49db-b02d-4519d1b5f3c9","Type":"ContainerStarted","Data":"1c8a01da12ce0ac09e0b982903e4ad718593a026ba4a3eda09e64def9a9268cb"}
Oct 30 22:15:26.346079 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:26.345958    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-0ce563-f9b586484-rg56t"
Oct 30 22:15:26.347689 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:26.347659    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0ce563-6b8c6c95bc-fsrrg" event={"ID":"74b993cb-2ab2-4457-8885-a2ee906a351f","Type":"ContainerStarted","Data":"eb147f6cf6260f99ff52e8729f7c9d9d9cdd4aa0f870a000a20859a65551fbe1"}
Oct 30 22:15:26.347805 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:26.347703    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0ce563-6b8c6c95bc-fsrrg" event={"ID":"74b993cb-2ab2-4457-8885-a2ee906a351f","Type":"ContainerStarted","Data":"68e4f4f2ac1cdfcf225de3574deef114ff450f5ab6e6bfb42ced8015deae27d7"}
Oct 30 22:15:26.347805 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:26.347722    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0ce563-6b8c6c95bc-fsrrg"
Oct 30 22:15:26.459014 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:26.458963    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-0ce563-f9b586484-rg56t" podStartSLOduration=1.458949474 podStartE2EDuration="1.458949474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:26.455994789 +0000 UTC m=+1980.068764576" watchObservedRunningTime="2026-10-30 22:15:26.458949474 +0000 UTC m=+1980.071719231"
Oct 30 22:15:26.576307 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:26.575683    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0ce563-6b8c6c95bc-fsrrg" podStartSLOduration=1.575663671 podStartE2EDuration="1.575663671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:26.569578804 +0000 UTC m=+1980.182348581" watchObservedRunningTime="2026-10-30 22:15:26.575663671 +0000 UTC m=+1980.188433429"
Oct 30 22:15:27.063179 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:27.063147    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-c1f64a-openshift-default-7d9c46ff4f-pnrz7"]
Oct 30 22:15:27.067343 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:27.067323    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c1f64a-openshift-default-7d9c46ff4f-pnrz7"
Oct 30 22:15:27.126559 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:27.124627    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-c1f64a-openshift-default-dockercfg-h9v7b\""
Oct 30 22:15:27.168551 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:27.165757    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cf14933-e261-4155-8a97-1a1a707c3550-istiod-ca-cert\") pod \"e2e-inf-a-c1f64a-openshift-default-7d9c46ff4f-pnrz7\" (UID: \"2cf14933-e261-4155-8a97-1a1a707c3550\") " pod="openshift-ingress/e2e-inf-a-c1f64a-openshift-default-7d9c46ff4f-pnrz7"
Oct 30 22:15:27.168551 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:27.165842    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cf14933-e261-4155-8a97-1a1a707c3550-workload-certs\") pod \"e2e-inf-a-c1f64a-openshift-default-7d9c46ff4f-pnrz7\" (UID: \"2cf14933-e261-4155-8a97-1a1a707c3550\") " pod="openshift-ingress/e2e-inf-a-c1f64a-openshift-default-7d9c46ff4f-pnrz7"
Oct 30 22:15:27.168551 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:27.165887    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zz5h6\" (UniqueName: \"kubernetes.io/projected/2cf14933-e261-4155-8a97-1a1a707c3550-kube-api-access-zz5h6\") pod \"e2e-inf-a-c1f64a-openshift-default-7d9c46ff4f-pnrz7\" (UID: \"2cf14933-e261-4155-8a97-1a1a707c3550\") " pod="openshift-ingress/e2e-inf-a-c1f64a-openshift-default-7d9c46ff4f-pnrz7"
Oct 30 22:15:27.168551 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:27.166007    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cf14933-e261-4155-8a97-1a1a707c3550-istio-envoy\") pod \"e2e-inf-a-c1f64a-openshift-default-7d9c46ff4f-pnrz7\" (UID: \"2cf14933-e261-4155-8a97-1a1a707c3550\") " pod="openshift-ingress/e2e-inf-a-c1f64a-openshift-default-7d9c46ff4f-pnrz7"
Oct 30 22:15:27.168551 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:27.166065    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cf14933-e261-4155-8a97-1a1a707c3550-credential-socket\") pod \"e2e-inf-a-c1f64a-openshift-default-7d9c46ff4f-pnrz7\" (UID: \"2cf14933-e261-4155-8a97-1a1a707c3550\") " pod="openshift-ingress/e2e-inf-a-c1f64a-openshift-default-7d9c46ff4f-pnrz7"
Oct 30 22:15:27.168551 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:27.166098    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cf14933-e261-4155-8a97-1a1a707c3550-istio-data\") pod \"e2e-inf-a-c1f64a-openshift-default-7d9c46ff4f-pnrz7\" (UID: \"2cf14933-e261-4155-8a97-1a1a707c3550\") " pod="openshift-ingress/e2e-inf-a-c1f64a-openshift-default-7d9c46ff4f-pnrz7"
Oct 30 22:15:27.168551 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:27.166139    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cf14933-e261-4155-8a97-1a1a707c3550-workload-socket\") pod \"e2e-inf-a-c1f64a-openshift-default-7d9c46ff4f-pnrz7\" (UID: \"2cf14933-e261-4155-8a97-1a1a707c3550\") " pod="openshift-ingress/e2e-inf-a-c1f64a-openshift-default-7d9c46ff4f-pnrz7"
Oct 30 22:15:27.168551 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:27.166168    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2cf14933-e261-4155-8a97-1a1a707c3550-istio-token\") pod \"e2e-inf-a-c1f64a-openshift-default-7d9c46ff4f-pnrz7\" (UID: \"2cf14933-e261-4155-8a97-1a1a707c3550\") " pod="openshift-ingress/e2e-inf-a-c1f64a-openshift-default-7d9c46ff4f-pnrz7"
Oct 30 22:15:27.168551 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:27.166306    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cf14933-e261-4155-8a97-1a1a707c3550-istio-podinfo\") pod \"e2e-inf-a-c1f64a-openshift-default-7d9c46ff4f-pnrz7\" (UID: \"2cf14933-e261-4155-8a97-1a1a707c3550\") " pod="openshift-ingress/e2e-inf-a-c1f64a-openshift-default-7d9c46ff4f-pnrz7"
Oct 30 22:15:27.261552 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:27.260700    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-c1f64a-openshift-default-7d9c46ff4f-pnrz7"]
Oct 30 22:15:27.272552 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:27.272492    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cf14933-e261-4155-8a97-1a1a707c3550-istiod-ca-cert\") pod \"e2e-inf-a-c1f64a-openshift-default-7d9c46ff4f-pnrz7\" (UID: \"2cf14933-e261-4155-8a97-1a1a707c3550\") " pod="openshift-ingress/e2e-inf-a-c1f64a-openshift-default-7d9c46ff4f-pnrz7"
Oct 30 22:15:27.272696 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:27.272591    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cf14933-e261-4155-8a97-1a1a707c3550-workload-certs\") pod \"e2e-inf-a-c1f64a-openshift-default-7d9c46ff4f-pnrz7\" (UID: \"2cf14933-e261-4155-8a97-1a1a707c3550\") " pod="openshift-ingress/e2e-inf-a-c1f64a-openshift-default-7d9c46ff4f-pnrz7"
Oct 30 22:15:27.272696 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:27.272632    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zz5h6\" (UniqueName: \"kubernetes.io/projected/2cf14933-e261-4155-8a97-1a1a707c3550-kube-api-access-zz5h6\") pod \"e2e-inf-a-c1f64a-openshift-default-7d9c46ff4f-pnrz7\" (UID: \"2cf14933-e261-4155-8a97-1a1a707c3550\") " pod="openshift-ingress/e2e-inf-a-c1f64a-openshift-default-7d9c46ff4f-pnrz7"
Oct 30 22:15:27.272797 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:27.272756    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cf14933-e261-4155-8a97-1a1a707c3550-istio-envoy\") pod \"e2e-inf-a-c1f64a-openshift-default-7d9c46ff4f-pnrz7\" (UID: \"2cf14933-e261-4155-8a97-1a1a707c3550\") " pod="openshift-ingress/e2e-inf-a-c1f64a-openshift-default-7d9c46ff4f-pnrz7"
Oct 30 22:15:27.272842 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:27.272814    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cf14933-e261-4155-8a97-1a1a707c3550-credential-socket\") pod \"e2e-inf-a-c1f64a-openshift-default-7d9c46ff4f-pnrz7\" (UID: \"2cf14933-e261-4155-8a97-1a1a707c3550\") " pod="openshift-ingress/e2e-inf-a-c1f64a-openshift-default-7d9c46ff4f-pnrz7"
Oct 30 22:15:27.272880 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:27.272848    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cf14933-e261-4155-8a97-1a1a707c3550-istio-data\") pod \"e2e-inf-a-c1f64a-openshift-default-7d9c46ff4f-pnrz7\" (UID: \"2cf14933-e261-4155-8a97-1a1a707c3550\") " pod="openshift-ingress/e2e-inf-a-c1f64a-openshift-default-7d9c46ff4f-pnrz7"
Oct 30 22:15:27.272917 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:27.272889    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cf14933-e261-4155-8a97-1a1a707c3550-workload-socket\") pod \"e2e-inf-a-c1f64a-openshift-default-7d9c46ff4f-pnrz7\" (UID: \"2cf14933-e261-4155-8a97-1a1a707c3550\") " pod="openshift-ingress/e2e-inf-a-c1f64a-openshift-default-7d9c46ff4f-pnrz7"
Oct 30 22:15:27.272954 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:27.272918    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2cf14933-e261-4155-8a97-1a1a707c3550-istio-token\") pod \"e2e-inf-a-c1f64a-openshift-default-7d9c46ff4f-pnrz7\" (UID: \"2cf14933-e261-4155-8a97-1a1a707c3550\") " pod="openshift-ingress/e2e-inf-a-c1f64a-openshift-default-7d9c46ff4f-pnrz7"
Oct 30 22:15:27.273064 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:27.273049    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cf14933-e261-4155-8a97-1a1a707c3550-istio-podinfo\") pod \"e2e-inf-a-c1f64a-openshift-default-7d9c46ff4f-pnrz7\" (UID: \"2cf14933-e261-4155-8a97-1a1a707c3550\") " pod="openshift-ingress/e2e-inf-a-c1f64a-openshift-default-7d9c46ff4f-pnrz7"
Oct 30 22:15:27.284254 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:27.274198    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cf14933-e261-4155-8a97-1a1a707c3550-workload-certs\") pod \"e2e-inf-a-c1f64a-openshift-default-7d9c46ff4f-pnrz7\" (UID: \"2cf14933-e261-4155-8a97-1a1a707c3550\") " pod="openshift-ingress/e2e-inf-a-c1f64a-openshift-default-7d9c46ff4f-pnrz7"
Oct 30 22:15:27.284254 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:27.274867    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cf14933-e261-4155-8a97-1a1a707c3550-istiod-ca-cert\") pod \"e2e-inf-a-c1f64a-openshift-default-7d9c46ff4f-pnrz7\" (UID: \"2cf14933-e261-4155-8a97-1a1a707c3550\") " pod="openshift-ingress/e2e-inf-a-c1f64a-openshift-default-7d9c46ff4f-pnrz7"
Oct 30 22:15:27.284254 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:27.275399    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cf14933-e261-4155-8a97-1a1a707c3550-istio-data\") pod \"e2e-inf-a-c1f64a-openshift-default-7d9c46ff4f-pnrz7\" (UID: \"2cf14933-e261-4155-8a97-1a1a707c3550\") " pod="openshift-ingress/e2e-inf-a-c1f64a-openshift-default-7d9c46ff4f-pnrz7"
Oct 30 22:15:27.284254 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:27.275661    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cf14933-e261-4155-8a97-1a1a707c3550-credential-socket\") pod \"e2e-inf-a-c1f64a-openshift-default-7d9c46ff4f-pnrz7\" (UID: \"2cf14933-e261-4155-8a97-1a1a707c3550\") " pod="openshift-ingress/e2e-inf-a-c1f64a-openshift-default-7d9c46ff4f-pnrz7"
Oct 30 22:15:27.284254 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:27.275901    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cf14933-e261-4155-8a97-1a1a707c3550-workload-socket\") pod \"e2e-inf-a-c1f64a-openshift-default-7d9c46ff4f-pnrz7\" (UID: \"2cf14933-e261-4155-8a97-1a1a707c3550\") " pod="openshift-ingress/e2e-inf-a-c1f64a-openshift-default-7d9c46ff4f-pnrz7"
Oct 30 22:15:27.284254 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:27.282005    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cf14933-e261-4155-8a97-1a1a707c3550-istio-envoy\") pod \"e2e-inf-a-c1f64a-openshift-default-7d9c46ff4f-pnrz7\" (UID: \"2cf14933-e261-4155-8a97-1a1a707c3550\") " pod="openshift-ingress/e2e-inf-a-c1f64a-openshift-default-7d9c46ff4f-pnrz7"
Oct 30 22:15:27.284254 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:27.282237    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cf14933-e261-4155-8a97-1a1a707c3550-istio-podinfo\") pod \"e2e-inf-a-c1f64a-openshift-default-7d9c46ff4f-pnrz7\" (UID: \"2cf14933-e261-4155-8a97-1a1a707c3550\") " pod="openshift-ingress/e2e-inf-a-c1f64a-openshift-default-7d9c46ff4f-pnrz7"
Oct 30 22:15:27.329395 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:27.329319    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zz5h6\" (UniqueName: \"kubernetes.io/projected/2cf14933-e261-4155-8a97-1a1a707c3550-kube-api-access-zz5h6\") pod \"e2e-inf-a-c1f64a-openshift-default-7d9c46ff4f-pnrz7\" (UID: \"2cf14933-e261-4155-8a97-1a1a707c3550\") " pod="openshift-ingress/e2e-inf-a-c1f64a-openshift-default-7d9c46ff4f-pnrz7"
Oct 30 22:15:27.329883 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:27.329844    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2cf14933-e261-4155-8a97-1a1a707c3550-istio-token\") pod \"e2e-inf-a-c1f64a-openshift-default-7d9c46ff4f-pnrz7\" (UID: \"2cf14933-e261-4155-8a97-1a1a707c3550\") " pod="openshift-ingress/e2e-inf-a-c1f64a-openshift-default-7d9c46ff4f-pnrz7"
Oct 30 22:15:27.383254 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:27.383221    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c1f64a-openshift-default-7d9c46ff4f-pnrz7"
Oct 30 22:15:27.768144 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:27.768061    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-263513-7c96866b48-56jwr"]
Oct 30 22:15:27.774317 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:27.774290    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-263513-7c96866b48-56jwr"
Oct 30 22:15:27.825540 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:27.825495    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-263513-serving-cert\""
Oct 30 22:15:27.829984 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:27.829954    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-263513-7c96866b48-56jwr"]
Oct 30 22:15:27.882854 ip-10-0-130-60 kubenswrapper[5061]: W1030 22:15:27.882819    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cf14933_e261_4155_8a97_1a1a707c3550.slice/crio-aa1bbc06374cf3a65ce5344fff990fa18549ba2ca842815659b48775ba1b3b88 WatchSource:0}: Error finding container aa1bbc06374cf3a65ce5344fff990fa18549ba2ca842815659b48775ba1b3b88: Status 404 returned error can't find the container with id aa1bbc06374cf3a65ce5344fff990fa18549ba2ca842815659b48775ba1b3b88
Oct 30 22:15:27.883658 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:27.883632    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49d78ffc-f174-411c-aae5-20b1c79f90d2-openshift-service-ca\") pod \"maas-api-e2e-inf-a-263513-7c96866b48-56jwr\" (UID: \"49d78ffc-f174-411c-aae5-20b1c79f90d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-263513-7c96866b48-56jwr"
Oct 30 22:15:27.883787 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:27.883760    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/49d78ffc-f174-411c-aae5-20b1c79f90d2-metrics-certs\") pod \"maas-api-e2e-inf-a-263513-7c96866b48-56jwr\" (UID: \"49d78ffc-f174-411c-aae5-20b1c79f90d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-263513-7c96866b48-56jwr"
Oct 30 22:15:27.884057 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:27.884039    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49d78ffc-f174-411c-aae5-20b1c79f90d2-maas-api-tls\") pod \"maas-api-e2e-inf-a-263513-7c96866b48-56jwr\" (UID: \"49d78ffc-f174-411c-aae5-20b1c79f90d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-263513-7c96866b48-56jwr"
Oct 30 22:15:27.884125 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:27.884112    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-877f9\" (UniqueName: \"kubernetes.io/projected/49d78ffc-f174-411c-aae5-20b1c79f90d2-kube-api-access-877f9\") pod \"maas-api-e2e-inf-a-263513-7c96866b48-56jwr\" (UID: \"49d78ffc-f174-411c-aae5-20b1c79f90d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-263513-7c96866b48-56jwr"
Oct 30 22:15:27.887034 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:27.886999    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:15:27.887180 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:27.887085    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:15:27.887180 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:27.887131    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:15:27.889860 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:27.889836    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-c1f64a-openshift-default-7d9c46ff4f-pnrz7"]
Oct 30 22:15:27.985622 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:27.985592    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49d78ffc-f174-411c-aae5-20b1c79f90d2-openshift-service-ca\") pod \"maas-api-e2e-inf-a-263513-7c96866b48-56jwr\" (UID: \"49d78ffc-f174-411c-aae5-20b1c79f90d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-263513-7c96866b48-56jwr"
Oct 30 22:15:27.985750 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:27.985684    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/49d78ffc-f174-411c-aae5-20b1c79f90d2-metrics-certs\") pod \"maas-api-e2e-inf-a-263513-7c96866b48-56jwr\" (UID: \"49d78ffc-f174-411c-aae5-20b1c79f90d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-263513-7c96866b48-56jwr"
Oct 30 22:15:27.985983 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:27.985964    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49d78ffc-f174-411c-aae5-20b1c79f90d2-maas-api-tls\") pod \"maas-api-e2e-inf-a-263513-7c96866b48-56jwr\" (UID: \"49d78ffc-f174-411c-aae5-20b1c79f90d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-263513-7c96866b48-56jwr"
Oct 30 22:15:27.986069 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:27.986051    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-877f9\" (UniqueName: \"kubernetes.io/projected/49d78ffc-f174-411c-aae5-20b1c79f90d2-kube-api-access-877f9\") pod \"maas-api-e2e-inf-a-263513-7c96866b48-56jwr\" (UID: \"49d78ffc-f174-411c-aae5-20b1c79f90d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-263513-7c96866b48-56jwr"
Oct 30 22:15:27.986278 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:27.986252    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49d78ffc-f174-411c-aae5-20b1c79f90d2-openshift-service-ca\") pod \"maas-api-e2e-inf-a-263513-7c96866b48-56jwr\" (UID: \"49d78ffc-f174-411c-aae5-20b1c79f90d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-263513-7c96866b48-56jwr"
Oct 30 22:15:27.988168 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:27.988148    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/49d78ffc-f174-411c-aae5-20b1c79f90d2-metrics-certs\") pod \"maas-api-e2e-inf-a-263513-7c96866b48-56jwr\" (UID: \"49d78ffc-f174-411c-aae5-20b1c79f90d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-263513-7c96866b48-56jwr"
Oct 30 22:15:27.988446 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:27.988423    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49d78ffc-f174-411c-aae5-20b1c79f90d2-maas-api-tls\") pod \"maas-api-e2e-inf-a-263513-7c96866b48-56jwr\" (UID: \"49d78ffc-f174-411c-aae5-20b1c79f90d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-263513-7c96866b48-56jwr"
Oct 30 22:15:28.040866 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:28.040798    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-877f9\" (UniqueName: \"kubernetes.io/projected/49d78ffc-f174-411c-aae5-20b1c79f90d2-kube-api-access-877f9\") pod \"maas-api-e2e-inf-a-263513-7c96866b48-56jwr\" (UID: \"49d78ffc-f174-411c-aae5-20b1c79f90d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-263513-7c96866b48-56jwr"
Oct 30 22:15:28.089501 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:28.089469    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-263513-7c96866b48-56jwr"
Oct 30 22:15:28.364754 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:28.364721    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c1f64a-openshift-default-7d9c46ff4f-pnrz7" event={"ID":"2cf14933-e261-4155-8a97-1a1a707c3550","Type":"ContainerStarted","Data":"aabf3da76eb49cc69fbbc7a4c143e478fb9db6b1c977cd981438852e705206a5"}
Oct 30 22:15:28.364875 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:28.364773    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c1f64a-openshift-default-7d9c46ff4f-pnrz7" event={"ID":"2cf14933-e261-4155-8a97-1a1a707c3550","Type":"ContainerStarted","Data":"aa1bbc06374cf3a65ce5344fff990fa18549ba2ca842815659b48775ba1b3b88"}
Oct 30 22:15:28.383428 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:28.383403    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-c1f64a-openshift-default-7d9c46ff4f-pnrz7"
Oct 30 22:15:28.385816 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:28.385794    5061 patch_prober.go:29] interesting pod/e2e-inf-a-c1f64a-openshift-default-7d9c46ff4f-pnrz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body=
Oct 30 22:15:28.385889 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:28.385843    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c1f64a-openshift-default-7d9c46ff4f-pnrz7" podUID="2cf14933-e261-4155-8a97-1a1a707c3550" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused"
Oct 30 22:15:28.478330 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:28.478273    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-c1f64a-openshift-default-7d9c46ff4f-pnrz7" podStartSLOduration=1.4782560980000001 podStartE2EDuration="1.478256098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:28.476374264 +0000 UTC m=+1982.089144033" watchObservedRunningTime="2026-10-30 22:15:28.478256098 +0000 UTC m=+1982.091025856"
Oct 30 22:15:28.494898 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:28.494850    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-263513-7c96866b48-56jwr"]
Oct 30 22:15:28.495323 ip-10-0-130-60 kubenswrapper[5061]: W1030 22:15:28.495300    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49d78ffc_f174_411c_aae5_20b1c79f90d2.slice/crio-e07b88be84f7cbd00144118818f16a40c49794dc8c00487d0a2c107c20afd6f9 WatchSource:0}: Error finding container e07b88be84f7cbd00144118818f16a40c49794dc8c00487d0a2c107c20afd6f9: Status 404 returned error can't find the container with id e07b88be84f7cbd00144118818f16a40c49794dc8c00487d0a2c107c20afd6f9
Oct 30 22:15:28.793894 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:28.793812    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-4rghz"]
Oct 30 22:15:28.794027 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:28.794003    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-4rghz" podUID="b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a" containerName="istio-proxy" containerID="cri-o://d85286201179d3d2462df7f6603001b24a7185aa408d0ed8c143a442a4c81d0c" gracePeriod=30
Oct 30 22:15:29.373981 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:29.373945    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-263513-7c96866b48-56jwr" event={"ID":"49d78ffc-f174-411c-aae5-20b1c79f90d2","Type":"ContainerStarted","Data":"e07b88be84f7cbd00144118818f16a40c49794dc8c00487d0a2c107c20afd6f9"}
Oct 30 22:15:29.388639 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:29.388610    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-c1f64a-openshift-default-7d9c46ff4f-pnrz7"
Oct 30 22:15:30.289338 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:30.289295    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-778d39d3-5cc665b758-td28d"]
Oct 30 22:15:30.383147 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:30.383113    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-263513-7c96866b48-56jwr" event={"ID":"49d78ffc-f174-411c-aae5-20b1c79f90d2","Type":"ContainerStarted","Data":"780fe6097849a8777aaed56649eec3c84f863676a9ea042c847062b87bc8bf0b"}
Oct 30 22:15:30.383504 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:30.383334    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-263513-7c96866b48-56jwr"
Oct 30 22:15:30.383504 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:30.383381    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-c1f64a-openshift-default-7d9c46ff4f-pnrz7"
Oct 30 22:15:30.384185 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:30.384161    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-c1f64a-openshift-default-7d9c46ff4f-pnrz7"
Oct 30 22:15:30.624386 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:30.624331    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-263513-7c96866b48-56jwr" podStartSLOduration=2.693861021 podStartE2EDuration="3.624318439s" totalImagesPullingTime="930.457418ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:30.622661254 +0000 UTC m=+1984.235431021" watchObservedRunningTime="2026-10-30 22:15:30.624318439 +0000 UTC m=+1984.237088202"
Oct 30 22:15:31.623816 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:31.623778    5061 patch_prober.go:29] interesting pod/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-4rghz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:15:31.624196 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:31.623829    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-4rghz" podUID="b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:15:34.088911 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:34.088878    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-4rghz"
Oct 30 22:15:34.203422 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:34.203390    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a-istiod-ca-cert\") pod \"b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a\" (UID: \"b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a\") "
Oct 30 22:15:34.203634 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:34.203463    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a-istio-token\" (UniqueName: \"kubernetes.io/projected/b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a-istio-token\") pod \"b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a\" (UID: \"b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a\") "
Oct 30 22:15:34.203634 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:34.203562    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a-credential-socket\") pod \"b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a\" (UID: \"b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a\") "
Oct 30 22:15:34.203753 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:34.203703    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a-kube-api-access-b56ms\" (UniqueName: \"kubernetes.io/projected/b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a-kube-api-access-b56ms\") pod \"b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a\" (UID: \"b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a\") "
Oct 30 22:15:34.203809 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:34.203772    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a-istio-data\") pod \"b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a\" (UID: \"b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a\") "
Oct 30 22:15:34.203859 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:34.203829    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a-workload-certs\") pod \"b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a\" (UID: \"b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a\") "
Oct 30 22:15:34.203910 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:34.203875    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a-credential-socket" pod "b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a" (UID: "b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:15:34.203960 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:34.203904    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a-istiod-ca-cert" pod "b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a" (UID: "b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:15:34.203960 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:34.203938    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a-workload-socket\") pod \"b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a\" (UID: \"b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a\") "
Oct 30 22:15:34.204054 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:34.204041    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a-istio-envoy\") pod \"b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a\" (UID: \"b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a\") "
Oct 30 22:15:34.204106 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:34.204093    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a-istio-podinfo\") pod \"b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a\" (UID: \"b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a\") "
Oct 30 22:15:34.204197 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:34.204090    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a-workload-certs" pod "b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a" (UID: "b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:15:34.204250 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:34.204178    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a-workload-socket" pod "b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a" (UID: "b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:15:34.204327 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:34.204019    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a-istio-data" pod "b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a" (UID: "b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:15:34.205176 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:34.205150    5061 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a-credential-socket\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:34.205344 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:34.205325    5061 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a-istio-data\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:34.205424 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:34.205356    5061 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a-workload-certs\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:34.205424 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:34.205373    5061 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a-workload-socket\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:34.205424 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:34.205390    5061 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a-istiod-ca-cert\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:34.206583 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:34.206515    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a-istio-token" pod "b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a" (UID: "b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:34.209630 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:34.209601    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a-kube-api-access-b56ms" pod "b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a" (UID: "b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a"). InnerVolumeSpecName "kube-api-access-b56ms". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:34.218650 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:34.218620    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a-istio-envoy" pod "b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a" (UID: "b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:15:34.218767 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:34.218671    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a-istio-podinfo" pod "b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a" (UID: "b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:15:34.307352 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:34.307319    5061 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a-istio-envoy\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:34.307352 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:34.307351    5061 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a-istio-podinfo\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:34.307352 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:34.307361    5061 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a-istio-token\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:34.307635 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:34.307371    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b56ms\" (UniqueName: \"kubernetes.io/projected/b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a-kube-api-access-b56ms\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:34.414034 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:34.413948    5061 generic.go:356] "Generic (PLEG): container finished" podID="b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a" containerID="d85286201179d3d2462df7f6603001b24a7185aa408d0ed8c143a442a4c81d0c" exitCode=0
Oct 30 22:15:34.414034 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:34.414007    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-4rghz"
Oct 30 22:15:34.414270 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:34.414028    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-4rghz" event={"ID":"b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a","Type":"ContainerDied","Data":"d85286201179d3d2462df7f6603001b24a7185aa408d0ed8c143a442a4c81d0c"}
Oct 30 22:15:34.414270 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:34.414111    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-4rghz" event={"ID":"b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a","Type":"ContainerDied","Data":"a00343b7f1e0db39817bbda59a5da5b7cc895484a454b254942cce3374534d47"}
Oct 30 22:15:34.414270 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:34.414136    5061 scope.go:122] "RemoveContainer" containerID="d85286201179d3d2462df7f6603001b24a7185aa408d0ed8c143a442a4c81d0c"
Oct 30 22:15:34.426169 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:34.426149    5061 scope.go:122] "RemoveContainer" containerID="d85286201179d3d2462df7f6603001b24a7185aa408d0ed8c143a442a4c81d0c"
Oct 30 22:15:34.426464 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:15:34.426445    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d85286201179d3d2462df7f6603001b24a7185aa408d0ed8c143a442a4c81d0c\": container with ID starting with d85286201179d3d2462df7f6603001b24a7185aa408d0ed8c143a442a4c81d0c not found: ID does not exist" containerID="d85286201179d3d2462df7f6603001b24a7185aa408d0ed8c143a442a4c81d0c"
Oct 30 22:15:34.426515 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:34.426472    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d85286201179d3d2462df7f6603001b24a7185aa408d0ed8c143a442a4c81d0c"} err="failed to get container status \"d85286201179d3d2462df7f6603001b24a7185aa408d0ed8c143a442a4c81d0c\": rpc error: code = NotFound desc = could not find container \"d85286201179d3d2462df7f6603001b24a7185aa408d0ed8c143a442a4c81d0c\": container with ID starting with d85286201179d3d2462df7f6603001b24a7185aa408d0ed8c143a442a4c81d0c not found: ID does not exist"
Oct 30 22:15:34.570622 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:34.570590    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-4rghz"]
Oct 30 22:15:34.576960 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:34.576927    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-4rghz"]
Oct 30 22:15:35.289997 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:35.289953    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-778d39d3-5cc665b758-td28d" podUID="b0abd91d-7465-4e7f-a840-827ddf311ce7" containerName="payload-processing" containerID="cri-o://f852069c4c6d8bce367216eb0e45333a23125648e02c584dbc7e2da2f868c00f" gracePeriod=25
Oct 30 22:15:36.397015 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:36.396988    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-263513-7c96866b48-56jwr"
Oct 30 22:15:36.534596 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:36.534564    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a" path="/var/lib/kubelet/pods/b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a/volumes"
Oct 30 22:15:37.354514 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:37.354480    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-446040-hwqrd"]
Oct 30 22:15:37.355254 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:37.355234    5061 cpu_manager.go:414] "containerMap: removing container" podUID="b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a" containerName="istio-proxy"
Oct 30 22:15:37.355342 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:37.355255    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a" containerName="istio-proxy"
Oct 30 22:15:37.355396 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:37.355370    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3ba1489-f9d8-4325-a1ac-65bc83ce4c0a" containerName="istio-proxy"
Oct 30 22:15:37.360667 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:37.360642    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0ce563-6b8c6c95bc-fsrrg"
Oct 30 22:15:37.360774 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:37.360687    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-0ce563-f9b586484-rg56t"
Oct 30 22:15:37.360774 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:37.360741    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-446040-hwqrd"
Oct 30 22:15:37.454520 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:37.454488    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5h94\" (UniqueName: \"kubernetes.io/projected/f9ec2d20-2901-4be4-8b81-0fb125cb4025-kube-api-access-h5h94\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-446040-hwqrd\" (UID: \"f9ec2d20-2901-4be4-8b81-0fb125cb4025\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-446040-hwqrd"
Oct 30 22:15:37.454983 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:37.454674    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9ec2d20-2901-4be4-8b81-0fb125cb4025-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-446040-hwqrd\" (UID: \"f9ec2d20-2901-4be4-8b81-0fb125cb4025\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-446040-hwqrd"
Oct 30 22:15:37.454983 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:37.454734    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9ec2d20-2901-4be4-8b81-0fb125cb4025-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-446040-hwqrd\" (UID: \"f9ec2d20-2901-4be4-8b81-0fb125cb4025\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-446040-hwqrd"
Oct 30 22:15:37.456749 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:37.456728    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gdmfg\""
Oct 30 22:15:37.456918 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:37.456730    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:15:37.460859 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:37.460831    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-446040-hwqrd"]
Oct 30 22:15:37.556904 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:37.556867    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5h94\" (UniqueName: \"kubernetes.io/projected/f9ec2d20-2901-4be4-8b81-0fb125cb4025-kube-api-access-h5h94\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-446040-hwqrd\" (UID: \"f9ec2d20-2901-4be4-8b81-0fb125cb4025\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-446040-hwqrd"
Oct 30 22:15:37.557067 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:37.556975    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9ec2d20-2901-4be4-8b81-0fb125cb4025-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-446040-hwqrd\" (UID: \"f9ec2d20-2901-4be4-8b81-0fb125cb4025\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-446040-hwqrd"
Oct 30 22:15:37.557067 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:37.557014    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9ec2d20-2901-4be4-8b81-0fb125cb4025-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-446040-hwqrd\" (UID: \"f9ec2d20-2901-4be4-8b81-0fb125cb4025\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-446040-hwqrd"
Oct 30 22:15:37.557792 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:37.557767    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9ec2d20-2901-4be4-8b81-0fb125cb4025-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-446040-hwqrd\" (UID: \"f9ec2d20-2901-4be4-8b81-0fb125cb4025\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-446040-hwqrd"
Oct 30 22:15:37.559366 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:37.559340    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9ec2d20-2901-4be4-8b81-0fb125cb4025-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-446040-hwqrd\" (UID: \"f9ec2d20-2901-4be4-8b81-0fb125cb4025\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-446040-hwqrd"
Oct 30 22:15:37.611520 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:37.611452    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5h94\" (UniqueName: \"kubernetes.io/projected/f9ec2d20-2901-4be4-8b81-0fb125cb4025-kube-api-access-h5h94\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-446040-hwqrd\" (UID: \"f9ec2d20-2901-4be4-8b81-0fb125cb4025\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-446040-hwqrd"
Oct 30 22:15:37.702818 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:37.702782    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-446040-hwqrd"
Oct 30 22:15:38.081454 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:38.081410    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-446040-hwqrd"]
Oct 30 22:15:38.081905 ip-10-0-130-60 kubenswrapper[5061]: W1030 22:15:38.081876    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9ec2d20_2901_4be4_8b81_0fb125cb4025.slice/crio-2ec7bfca0320b87fbf2eadb689f2e1c9f3268cad1d09ad0d3ebf4f60eeb829a8 WatchSource:0}: Error finding container 2ec7bfca0320b87fbf2eadb689f2e1c9f3268cad1d09ad0d3ebf4f60eeb829a8: Status 404 returned error can't find the container with id 2ec7bfca0320b87fbf2eadb689f2e1c9f3268cad1d09ad0d3ebf4f60eeb829a8
Oct 30 22:15:38.452199 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:38.452166    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-446040-hwqrd" event={"ID":"f9ec2d20-2901-4be4-8b81-0fb125cb4025","Type":"ContainerStarted","Data":"2ec7bfca0320b87fbf2eadb689f2e1c9f3268cad1d09ad0d3ebf4f60eeb829a8"}
Oct 30 22:15:39.461161 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:39.461132    5061 generic.go:356] "Generic (PLEG): container finished" podID="f9ec2d20-2901-4be4-8b81-0fb125cb4025" containerID="1c31084efb8058bc104bc1c2190660c8bba6910951b0afba1a9792d0cabe57ad" exitCode=0
Oct 30 22:15:39.461518 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:39.461220    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-446040-hwqrd" event={"ID":"f9ec2d20-2901-4be4-8b81-0fb125cb4025","Type":"ContainerDied","Data":"1c31084efb8058bc104bc1c2190660c8bba6910951b0afba1a9792d0cabe57ad"}
Oct 30 22:15:40.663295 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:40.663262    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-446040-hwqrd"
Oct 30 22:15:40.728747 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:40.728665    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9ec2d20-2901-4be4-8b81-0fb125cb4025-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9ec2d20-2901-4be4-8b81-0fb125cb4025-maas-api-xks-ca\") pod \"f9ec2d20-2901-4be4-8b81-0fb125cb4025\" (UID: \"f9ec2d20-2901-4be4-8b81-0fb125cb4025\") "
Oct 30 22:15:40.728895 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:40.728862    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9ec2d20-2901-4be4-8b81-0fb125cb4025-kube-api-access-h5h94\" (UniqueName: \"kubernetes.io/projected/f9ec2d20-2901-4be4-8b81-0fb125cb4025-kube-api-access-h5h94\") pod \"f9ec2d20-2901-4be4-8b81-0fb125cb4025\" (UID: \"f9ec2d20-2901-4be4-8b81-0fb125cb4025\") "
Oct 30 22:15:40.728938 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:40.728911    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9ec2d20-2901-4be4-8b81-0fb125cb4025-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9ec2d20-2901-4be4-8b81-0fb125cb4025-maas-api-openshift-ca\") pod \"f9ec2d20-2901-4be4-8b81-0fb125cb4025\" (UID: \"f9ec2d20-2901-4be4-8b81-0fb125cb4025\") "
Oct 30 22:15:40.729335 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:40.729306    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9ec2d20-2901-4be4-8b81-0fb125cb4025-maas-api-openshift-ca" pod "f9ec2d20-2901-4be4-8b81-0fb125cb4025" (UID: "f9ec2d20-2901-4be4-8b81-0fb125cb4025"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:15:40.729450 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:40.729423    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9ec2d20-2901-4be4-8b81-0fb125cb4025-maas-api-openshift-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:40.730909 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:40.730882    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9ec2d20-2901-4be4-8b81-0fb125cb4025-maas-api-xks-ca" pod "f9ec2d20-2901-4be4-8b81-0fb125cb4025" (UID: "f9ec2d20-2901-4be4-8b81-0fb125cb4025"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:15:40.731004 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:40.730948    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9ec2d20-2901-4be4-8b81-0fb125cb4025-kube-api-access-h5h94" pod "f9ec2d20-2901-4be4-8b81-0fb125cb4025" (UID: "f9ec2d20-2901-4be4-8b81-0fb125cb4025"). InnerVolumeSpecName "kube-api-access-h5h94". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:40.830555 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:40.830496    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5h94\" (UniqueName: \"kubernetes.io/projected/f9ec2d20-2901-4be4-8b81-0fb125cb4025-kube-api-access-h5h94\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:40.830555 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:40.830557    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9ec2d20-2901-4be4-8b81-0fb125cb4025-maas-api-xks-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:41.476315 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:41.476285    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-446040-hwqrd"
Oct 30 22:15:41.476315 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:41.476304    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-446040-hwqrd" event={"ID":"f9ec2d20-2901-4be4-8b81-0fb125cb4025","Type":"ContainerDied","Data":"2ec7bfca0320b87fbf2eadb689f2e1c9f3268cad1d09ad0d3ebf4f60eeb829a8"}
Oct 30 22:15:41.476490 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:41.476355    5061 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2ec7bfca0320b87fbf2eadb689f2e1c9f3268cad1d09ad0d3ebf4f60eeb829a8"
Oct 30 22:15:43.865668 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:43.865631    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-778d39d3-5cc665b758-td28d" podUID="b0abd91d-7465-4e7f-a840-827ddf311ce7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:15:43.968773 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:43.968732    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-f95pk"]
Oct 30 22:15:43.968971 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:43.968940    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-f95pk" podUID="081e86b3-a830-4a91-a7da-c270baca6eee" containerName="istio-proxy" containerID="cri-o://7e00c8d518881ac6440d33fac0fff34bc736724b28ddab555df35b72d0214c02" gracePeriod=30
Oct 30 22:15:46.585437 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:46.585398    5061 patch_prober.go:29] interesting pod/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-f95pk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:15:46.585888 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:46.585454    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-f95pk" podUID="081e86b3-a830-4a91-a7da-c270baca6eee" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:15:49.278384 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:49.278358    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-f95pk"
Oct 30 22:15:49.352362 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:49.352324    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/081e86b3-a830-4a91-a7da-c270baca6eee-istio-token\" (UniqueName: \"kubernetes.io/projected/081e86b3-a830-4a91-a7da-c270baca6eee-istio-token\") pod \"081e86b3-a830-4a91-a7da-c270baca6eee\" (UID: \"081e86b3-a830-4a91-a7da-c270baca6eee\") "
Oct 30 22:15:49.352663 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:49.352641    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/081e86b3-a830-4a91-a7da-c270baca6eee-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/081e86b3-a830-4a91-a7da-c270baca6eee-credential-socket\") pod \"081e86b3-a830-4a91-a7da-c270baca6eee\" (UID: \"081e86b3-a830-4a91-a7da-c270baca6eee\") "
Oct 30 22:15:49.352880 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:49.352863    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/081e86b3-a830-4a91-a7da-c270baca6eee-kube-api-access-76jqn\" (UniqueName: \"kubernetes.io/projected/081e86b3-a830-4a91-a7da-c270baca6eee-kube-api-access-76jqn\") pod \"081e86b3-a830-4a91-a7da-c270baca6eee\" (UID: \"081e86b3-a830-4a91-a7da-c270baca6eee\") "
Oct 30 22:15:49.358928 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:49.358880    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/081e86b3-a830-4a91-a7da-c270baca6eee-credential-socket" pod "081e86b3-a830-4a91-a7da-c270baca6eee" (UID: "081e86b3-a830-4a91-a7da-c270baca6eee"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:15:49.359122 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:49.358917    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/081e86b3-a830-4a91-a7da-c270baca6eee-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/081e86b3-a830-4a91-a7da-c270baca6eee-istio-data\") pod \"081e86b3-a830-4a91-a7da-c270baca6eee\" (UID: \"081e86b3-a830-4a91-a7da-c270baca6eee\") "
Oct 30 22:15:49.359247 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:49.359232    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/081e86b3-a830-4a91-a7da-c270baca6eee-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/081e86b3-a830-4a91-a7da-c270baca6eee-istio-podinfo\") pod \"081e86b3-a830-4a91-a7da-c270baca6eee\" (UID: \"081e86b3-a830-4a91-a7da-c270baca6eee\") "
Oct 30 22:15:49.359367 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:49.359352    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/081e86b3-a830-4a91-a7da-c270baca6eee-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/081e86b3-a830-4a91-a7da-c270baca6eee-istiod-ca-cert\") pod \"081e86b3-a830-4a91-a7da-c270baca6eee\" (UID: \"081e86b3-a830-4a91-a7da-c270baca6eee\") "
Oct 30 22:15:49.359965 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:49.359648    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/081e86b3-a830-4a91-a7da-c270baca6eee-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/081e86b3-a830-4a91-a7da-c270baca6eee-istio-envoy\") pod \"081e86b3-a830-4a91-a7da-c270baca6eee\" (UID: \"081e86b3-a830-4a91-a7da-c270baca6eee\") "
Oct 30 22:15:49.360165 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:49.360143    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/081e86b3-a830-4a91-a7da-c270baca6eee-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/081e86b3-a830-4a91-a7da-c270baca6eee-workload-socket\") pod \"081e86b3-a830-4a91-a7da-c270baca6eee\" (UID: \"081e86b3-a830-4a91-a7da-c270baca6eee\") "
Oct 30 22:15:49.360342 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:49.360327    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/081e86b3-a830-4a91-a7da-c270baca6eee-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/081e86b3-a830-4a91-a7da-c270baca6eee-workload-certs\") pod \"081e86b3-a830-4a91-a7da-c270baca6eee\" (UID: \"081e86b3-a830-4a91-a7da-c270baca6eee\") "
Oct 30 22:15:49.361666 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:49.361640    5061 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/081e86b3-a830-4a91-a7da-c270baca6eee-credential-socket\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:49.373620 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:49.359422    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/081e86b3-a830-4a91-a7da-c270baca6eee-istio-data" pod "081e86b3-a830-4a91-a7da-c270baca6eee" (UID: "081e86b3-a830-4a91-a7da-c270baca6eee"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:15:49.373816 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:49.361920    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/081e86b3-a830-4a91-a7da-c270baca6eee-istiod-ca-cert" pod "081e86b3-a830-4a91-a7da-c270baca6eee" (UID: "081e86b3-a830-4a91-a7da-c270baca6eee"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:15:49.373906 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:49.362778    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/081e86b3-a830-4a91-a7da-c270baca6eee-workload-certs" pod "081e86b3-a830-4a91-a7da-c270baca6eee" (UID: "081e86b3-a830-4a91-a7da-c270baca6eee"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:15:49.373985 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:49.363054    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/081e86b3-a830-4a91-a7da-c270baca6eee-istio-token" pod "081e86b3-a830-4a91-a7da-c270baca6eee" (UID: "081e86b3-a830-4a91-a7da-c270baca6eee"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:49.374057 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:49.373552    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/081e86b3-a830-4a91-a7da-c270baca6eee-workload-socket" pod "081e86b3-a830-4a91-a7da-c270baca6eee" (UID: "081e86b3-a830-4a91-a7da-c270baca6eee"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:15:49.393564 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:49.390127    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/081e86b3-a830-4a91-a7da-c270baca6eee-kube-api-access-76jqn" pod "081e86b3-a830-4a91-a7da-c270baca6eee" (UID: "081e86b3-a830-4a91-a7da-c270baca6eee"). InnerVolumeSpecName "kube-api-access-76jqn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:49.393564 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:49.392162    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/081e86b3-a830-4a91-a7da-c270baca6eee-istio-podinfo" pod "081e86b3-a830-4a91-a7da-c270baca6eee" (UID: "081e86b3-a830-4a91-a7da-c270baca6eee"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:15:49.402554 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:49.398898    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/081e86b3-a830-4a91-a7da-c270baca6eee-istio-envoy" pod "081e86b3-a830-4a91-a7da-c270baca6eee" (UID: "081e86b3-a830-4a91-a7da-c270baca6eee"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:15:49.475778 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:49.475733    5061 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/081e86b3-a830-4a91-a7da-c270baca6eee-istio-envoy\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:49.475778 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:49.475779    5061 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/081e86b3-a830-4a91-a7da-c270baca6eee-workload-socket\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:49.476038 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:49.475797    5061 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/081e86b3-a830-4a91-a7da-c270baca6eee-workload-certs\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:49.476038 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:49.475814    5061 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/081e86b3-a830-4a91-a7da-c270baca6eee-istio-token\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:49.476038 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:49.475828    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76jqn\" (UniqueName: \"kubernetes.io/projected/081e86b3-a830-4a91-a7da-c270baca6eee-kube-api-access-76jqn\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:49.476038 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:49.475845    5061 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/081e86b3-a830-4a91-a7da-c270baca6eee-istio-data\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:49.476038 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:49.475859    5061 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/081e86b3-a830-4a91-a7da-c270baca6eee-istio-podinfo\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:49.476038 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:49.475874    5061 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/081e86b3-a830-4a91-a7da-c270baca6eee-istiod-ca-cert\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:49.544048 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:49.543994    5061 generic.go:356] "Generic (PLEG): container finished" podID="081e86b3-a830-4a91-a7da-c270baca6eee" containerID="7e00c8d518881ac6440d33fac0fff34bc736724b28ddab555df35b72d0214c02" exitCode=0
Oct 30 22:15:49.544048 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:49.544036    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-f95pk" event={"ID":"081e86b3-a830-4a91-a7da-c270baca6eee","Type":"ContainerDied","Data":"7e00c8d518881ac6440d33fac0fff34bc736724b28ddab555df35b72d0214c02"}
Oct 30 22:15:49.544247 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:49.544068    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-f95pk"
Oct 30 22:15:49.544247 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:49.544086    5061 scope.go:122] "RemoveContainer" containerID="7e00c8d518881ac6440d33fac0fff34bc736724b28ddab555df35b72d0214c02"
Oct 30 22:15:49.544247 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:49.544077    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-f95pk" event={"ID":"081e86b3-a830-4a91-a7da-c270baca6eee","Type":"ContainerDied","Data":"be72527efbd5e8c7d6e2071db97dff9a6010498d3e2432ff89b1ba24302a46aa"}
Oct 30 22:15:49.556438 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:49.556340    5061 scope.go:122] "RemoveContainer" containerID="7e00c8d518881ac6440d33fac0fff34bc736724b28ddab555df35b72d0214c02"
Oct 30 22:15:49.556736 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:15:49.556710    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e00c8d518881ac6440d33fac0fff34bc736724b28ddab555df35b72d0214c02\": container with ID starting with 7e00c8d518881ac6440d33fac0fff34bc736724b28ddab555df35b72d0214c02 not found: ID does not exist" containerID="7e00c8d518881ac6440d33fac0fff34bc736724b28ddab555df35b72d0214c02"
Oct 30 22:15:49.556822 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:49.556743    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e00c8d518881ac6440d33fac0fff34bc736724b28ddab555df35b72d0214c02"} err="failed to get container status \"7e00c8d518881ac6440d33fac0fff34bc736724b28ddab555df35b72d0214c02\": rpc error: code = NotFound desc = could not find container \"7e00c8d518881ac6440d33fac0fff34bc736724b28ddab555df35b72d0214c02\": container with ID starting with 7e00c8d518881ac6440d33fac0fff34bc736724b28ddab555df35b72d0214c02 not found: ID does not exist"
Oct 30 22:15:49.723549 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:49.723037    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-f95pk"]
Oct 30 22:15:49.729583 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:49.728940    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-f95pk"]
Oct 30 22:15:50.675656 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:50.668939    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="081e86b3-a830-4a91-a7da-c270baca6eee" path="/var/lib/kubelet/pods/081e86b3-a830-4a91-a7da-c270baca6eee/volumes"
Oct 30 22:15:53.865895 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:53.865850    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-778d39d3-5cc665b758-td28d" podUID="b0abd91d-7465-4e7f-a840-827ddf311ce7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:15:54.669489 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:15:54.669453    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 22:15:55.640659 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:55.640301    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-778d39d3-5cc665b758-td28d"
Oct 30 22:15:55.693305 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:55.693266    5061 generic.go:356] "Generic (PLEG): container finished" podID="b0abd91d-7465-4e7f-a840-827ddf311ce7" containerID="f852069c4c6d8bce367216eb0e45333a23125648e02c584dbc7e2da2f868c00f" exitCode=0
Oct 30 22:15:55.693555 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:55.693313    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-778d39d3-5cc665b758-td28d"
Oct 30 22:15:55.693748 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:55.693330    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-778d39d3-5cc665b758-td28d" event={"ID":"b0abd91d-7465-4e7f-a840-827ddf311ce7","Type":"ContainerDied","Data":"f852069c4c6d8bce367216eb0e45333a23125648e02c584dbc7e2da2f868c00f"}
Oct 30 22:15:55.693834 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:55.693815    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-778d39d3-5cc665b758-td28d" event={"ID":"b0abd91d-7465-4e7f-a840-827ddf311ce7","Type":"ContainerDied","Data":"dc60efa1eb8fecc91e9e09dccc26c4a92585af62a34481165bd795fd4ca55b3f"}
Oct 30 22:15:55.693888 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:55.693851    5061 scope.go:122] "RemoveContainer" containerID="f852069c4c6d8bce367216eb0e45333a23125648e02c584dbc7e2da2f868c00f"
Oct 30 22:15:55.707085 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:55.707058    5061 scope.go:122] "RemoveContainer" containerID="f852069c4c6d8bce367216eb0e45333a23125648e02c584dbc7e2da2f868c00f"
Oct 30 22:15:55.707479 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:15:55.707456    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f852069c4c6d8bce367216eb0e45333a23125648e02c584dbc7e2da2f868c00f\": container with ID starting with f852069c4c6d8bce367216eb0e45333a23125648e02c584dbc7e2da2f868c00f not found: ID does not exist" containerID="f852069c4c6d8bce367216eb0e45333a23125648e02c584dbc7e2da2f868c00f"
Oct 30 22:15:55.707632 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:55.707494    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f852069c4c6d8bce367216eb0e45333a23125648e02c584dbc7e2da2f868c00f"} err="failed to get container status \"f852069c4c6d8bce367216eb0e45333a23125648e02c584dbc7e2da2f868c00f\": rpc error: code = NotFound desc = could not find container \"f852069c4c6d8bce367216eb0e45333a23125648e02c584dbc7e2da2f868c00f\": container with ID starting with f852069c4c6d8bce367216eb0e45333a23125648e02c584dbc7e2da2f868c00f not found: ID does not exist"
Oct 30 22:15:55.757431 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:55.757394    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0abd91d-7465-4e7f-a840-827ddf311ce7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0abd91d-7465-4e7f-a840-827ddf311ce7-plugins-config-volume\") pod \"b0abd91d-7465-4e7f-a840-827ddf311ce7\" (UID: \"b0abd91d-7465-4e7f-a840-827ddf311ce7\") "
Oct 30 22:15:55.758251 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:55.758189    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0abd91d-7465-4e7f-a840-827ddf311ce7-plugins-config-volume" pod "b0abd91d-7465-4e7f-a840-827ddf311ce7" (UID: "b0abd91d-7465-4e7f-a840-827ddf311ce7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:15:55.758401 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:55.758360    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0abd91d-7465-4e7f-a840-827ddf311ce7-kube-api-access-g9xtk\" (UniqueName: \"kubernetes.io/projected/b0abd91d-7465-4e7f-a840-827ddf311ce7-kube-api-access-g9xtk\") pod \"b0abd91d-7465-4e7f-a840-827ddf311ce7\" (UID: \"b0abd91d-7465-4e7f-a840-827ddf311ce7\") "
Oct 30 22:15:55.759377 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:55.759348    5061 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0abd91d-7465-4e7f-a840-827ddf311ce7-plugins-config-volume\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:55.762032 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:55.762006    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0abd91d-7465-4e7f-a840-827ddf311ce7-kube-api-access-g9xtk" pod "b0abd91d-7465-4e7f-a840-827ddf311ce7" (UID: "b0abd91d-7465-4e7f-a840-827ddf311ce7"). InnerVolumeSpecName "kube-api-access-g9xtk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:55.861457 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:55.861355    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9xtk\" (UniqueName: \"kubernetes.io/projected/b0abd91d-7465-4e7f-a840-827ddf311ce7-kube-api-access-g9xtk\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:56.114681 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:56.114574    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-778d39d3-5cc665b758-td28d"]
Oct 30 22:15:56.120492 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:56.120460    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-778d39d3-5cc665b758-td28d"]
Oct 30 22:15:56.532769 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:56.532682    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0abd91d-7465-4e7f-a840-827ddf311ce7" path="/var/lib/kubelet/pods/b0abd91d-7465-4e7f-a840-827ddf311ce7/volumes"
Oct 30 22:15:58.531220 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:58.531188    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-bpksk"]
Oct 30 22:15:58.531687 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:15:58.531410    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-bpksk" podUID="c88c84eb-d49e-49f8-b26e-429659b48ad6" containerName="istio-proxy" containerID="cri-o://e99ff9345b2bd8c8ff797cf09541c95857c43bb9d33cbf9c830df59f0b23e2fc" gracePeriod=30
Oct 30 22:16:00.037812 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:00.035545    5061 patch_prober.go:29] interesting pod/e2e-api-a-25de17-openshift-default-8687bbf6cc-p9mf7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": read tcp 10.132.0.2:37810->10.132.0.129:15021: read: connection reset by peer" start-of-body=
Oct 30 22:16:00.037812 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:00.035615    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-25de17-openshift-default-8687bbf6cc-p9mf7" podUID="f464e173-8ada-4978-967f-7897d5bc5823" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": read tcp 10.132.0.2:37810->10.132.0.129:15021: read: connection reset by peer"
Oct 30 22:16:00.738869 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:00.738833    5061 generic.go:356] "Generic (PLEG): container finished" podID="f464e173-8ada-4978-967f-7897d5bc5823" containerID="afb0f3ee46792d05f5b3ccc7c84b1d1ecdbe202f6943a3d166ff2be91f705a77" exitCode=0
Oct 30 22:16:00.739076 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:00.738903    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-25de17-openshift-default-8687bbf6cc-p9mf7" event={"ID":"f464e173-8ada-4978-967f-7897d5bc5823","Type":"ContainerDied","Data":"afb0f3ee46792d05f5b3ccc7c84b1d1ecdbe202f6943a3d166ff2be91f705a77"}
Oct 30 22:16:00.739280 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:00.739265    5061 scope.go:122] "RemoveContainer" containerID="afb0f3ee46792d05f5b3ccc7c84b1d1ecdbe202f6943a3d166ff2be91f705a77"
Oct 30 22:16:00.740904 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:00.740877    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:16:00.740980 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:00.740947    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:16:00.741018 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:00.740979    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:16:01.748722 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:01.748684    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-25de17-openshift-default-8687bbf6cc-p9mf7" event={"ID":"f464e173-8ada-4978-967f-7897d5bc5823","Type":"ContainerStarted","Data":"33a76dbb7759715346ee34e1de9d391b3bfc30b2dc6d18aa051fbbfe65bc7177"}
Oct 30 22:16:02.403840 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:02.403805    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-25de17-openshift-default-8687bbf6cc-p9mf7"
Oct 30 22:16:02.404033 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:02.403982    5061 patch_prober.go:29] interesting pod/e2e-api-a-25de17-openshift-default-8687bbf6cc-p9mf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body=
Oct 30 22:16:02.404086 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:02.404027    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-25de17-openshift-default-8687bbf6cc-p9mf7" podUID="f464e173-8ada-4978-967f-7897d5bc5823" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused"
Oct 30 22:16:03.404329 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:03.404291    5061 patch_prober.go:29] interesting pod/e2e-api-a-25de17-openshift-default-8687bbf6cc-p9mf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body=
Oct 30 22:16:03.404787 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:03.404343    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-25de17-openshift-default-8687bbf6cc-p9mf7" podUID="f464e173-8ada-4978-967f-7897d5bc5823" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused"
Oct 30 22:16:03.769309 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:03.769274    5061 generic.go:356] "Generic (PLEG): container finished" podID="c88c84eb-d49e-49f8-b26e-429659b48ad6" containerID="e99ff9345b2bd8c8ff797cf09541c95857c43bb9d33cbf9c830df59f0b23e2fc" exitCode=0
Oct 30 22:16:03.769469 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:03.769367    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-bpksk" event={"ID":"c88c84eb-d49e-49f8-b26e-429659b48ad6","Type":"ContainerDied","Data":"e99ff9345b2bd8c8ff797cf09541c95857c43bb9d33cbf9c830df59f0b23e2fc"}
Oct 30 22:16:03.864735 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:03.864711    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-bpksk"
Oct 30 22:16:03.987503 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:03.987419    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c88c84eb-d49e-49f8-b26e-429659b48ad6-istio-token\" (UniqueName: \"kubernetes.io/projected/c88c84eb-d49e-49f8-b26e-429659b48ad6-istio-token\") pod \"c88c84eb-d49e-49f8-b26e-429659b48ad6\" (UID: \"c88c84eb-d49e-49f8-b26e-429659b48ad6\") "
Oct 30 22:16:03.987503 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:03.987472    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c88c84eb-d49e-49f8-b26e-429659b48ad6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c88c84eb-d49e-49f8-b26e-429659b48ad6-istio-podinfo\") pod \"c88c84eb-d49e-49f8-b26e-429659b48ad6\" (UID: \"c88c84eb-d49e-49f8-b26e-429659b48ad6\") "
Oct 30 22:16:03.987741 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:03.987591    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c88c84eb-d49e-49f8-b26e-429659b48ad6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c88c84eb-d49e-49f8-b26e-429659b48ad6-workload-socket\") pod \"c88c84eb-d49e-49f8-b26e-429659b48ad6\" (UID: \"c88c84eb-d49e-49f8-b26e-429659b48ad6\") "
Oct 30 22:16:03.987741 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:03.987623    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c88c84eb-d49e-49f8-b26e-429659b48ad6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c88c84eb-d49e-49f8-b26e-429659b48ad6-istiod-ca-cert\") pod \"c88c84eb-d49e-49f8-b26e-429659b48ad6\" (UID: \"c88c84eb-d49e-49f8-b26e-429659b48ad6\") "
Oct 30 22:16:03.987741 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:03.987641    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c88c84eb-d49e-49f8-b26e-429659b48ad6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c88c84eb-d49e-49f8-b26e-429659b48ad6-istio-envoy\") pod \"c88c84eb-d49e-49f8-b26e-429659b48ad6\" (UID: \"c88c84eb-d49e-49f8-b26e-429659b48ad6\") "
Oct 30 22:16:03.987741 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:03.987720    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c88c84eb-d49e-49f8-b26e-429659b48ad6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c88c84eb-d49e-49f8-b26e-429659b48ad6-workload-certs\") pod \"c88c84eb-d49e-49f8-b26e-429659b48ad6\" (UID: \"c88c84eb-d49e-49f8-b26e-429659b48ad6\") "
Oct 30 22:16:03.987979 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:03.987833    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c88c84eb-d49e-49f8-b26e-429659b48ad6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c88c84eb-d49e-49f8-b26e-429659b48ad6-credential-socket\") pod \"c88c84eb-d49e-49f8-b26e-429659b48ad6\" (UID: \"c88c84eb-d49e-49f8-b26e-429659b48ad6\") "
Oct 30 22:16:03.987979 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:03.987867    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c88c84eb-d49e-49f8-b26e-429659b48ad6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c88c84eb-d49e-49f8-b26e-429659b48ad6-istio-data\") pod \"c88c84eb-d49e-49f8-b26e-429659b48ad6\" (UID: \"c88c84eb-d49e-49f8-b26e-429659b48ad6\") "
Oct 30 22:16:03.987979 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:03.987920    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c88c84eb-d49e-49f8-b26e-429659b48ad6-kube-api-access-nwg2v\" (UniqueName: \"kubernetes.io/projected/c88c84eb-d49e-49f8-b26e-429659b48ad6-kube-api-access-nwg2v\") pod \"c88c84eb-d49e-49f8-b26e-429659b48ad6\" (UID: \"c88c84eb-d49e-49f8-b26e-429659b48ad6\") "
Oct 30 22:16:03.988128 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:03.988072    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c88c84eb-d49e-49f8-b26e-429659b48ad6-workload-socket" pod "c88c84eb-d49e-49f8-b26e-429659b48ad6" (UID: "c88c84eb-d49e-49f8-b26e-429659b48ad6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:16:03.988128 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:03.988093    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c88c84eb-d49e-49f8-b26e-429659b48ad6-workload-certs" pod "c88c84eb-d49e-49f8-b26e-429659b48ad6" (UID: "c88c84eb-d49e-49f8-b26e-429659b48ad6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:16:03.988351 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:03.988328    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c88c84eb-d49e-49f8-b26e-429659b48ad6-credential-socket" pod "c88c84eb-d49e-49f8-b26e-429659b48ad6" (UID: "c88c84eb-d49e-49f8-b26e-429659b48ad6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:16:03.988565 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:03.988544    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c88c84eb-d49e-49f8-b26e-429659b48ad6-istio-data" pod "c88c84eb-d49e-49f8-b26e-429659b48ad6" (UID: "c88c84eb-d49e-49f8-b26e-429659b48ad6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:16:03.988750 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:03.988720    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c88c84eb-d49e-49f8-b26e-429659b48ad6-istiod-ca-cert" pod "c88c84eb-d49e-49f8-b26e-429659b48ad6" (UID: "c88c84eb-d49e-49f8-b26e-429659b48ad6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:16:03.988926 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:03.988906    5061 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c88c84eb-d49e-49f8-b26e-429659b48ad6-workload-socket\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:16:03.989023 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:03.988930    5061 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c88c84eb-d49e-49f8-b26e-429659b48ad6-istiod-ca-cert\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:16:03.989023 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:03.988942    5061 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c88c84eb-d49e-49f8-b26e-429659b48ad6-workload-certs\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:16:03.989023 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:03.988955    5061 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c88c84eb-d49e-49f8-b26e-429659b48ad6-credential-socket\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:16:03.989023 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:03.988967    5061 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c88c84eb-d49e-49f8-b26e-429659b48ad6-istio-data\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:16:03.989938 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:03.989917    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c88c84eb-d49e-49f8-b26e-429659b48ad6-istio-token" pod "c88c84eb-d49e-49f8-b26e-429659b48ad6" (UID: "c88c84eb-d49e-49f8-b26e-429659b48ad6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:16:03.990158 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:03.990137    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c88c84eb-d49e-49f8-b26e-429659b48ad6-istio-podinfo" pod "c88c84eb-d49e-49f8-b26e-429659b48ad6" (UID: "c88c84eb-d49e-49f8-b26e-429659b48ad6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:16:03.990158 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:03.990142    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c88c84eb-d49e-49f8-b26e-429659b48ad6-istio-envoy" pod "c88c84eb-d49e-49f8-b26e-429659b48ad6" (UID: "c88c84eb-d49e-49f8-b26e-429659b48ad6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:16:03.990301 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:03.990286    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c88c84eb-d49e-49f8-b26e-429659b48ad6-kube-api-access-nwg2v" pod "c88c84eb-d49e-49f8-b26e-429659b48ad6" (UID: "c88c84eb-d49e-49f8-b26e-429659b48ad6"). InnerVolumeSpecName "kube-api-access-nwg2v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:16:04.090340 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:04.090297    5061 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c88c84eb-d49e-49f8-b26e-429659b48ad6-istio-token\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:16:04.090492 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:04.090344    5061 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c88c84eb-d49e-49f8-b26e-429659b48ad6-istio-podinfo\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:16:04.090492 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:04.090361    5061 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c88c84eb-d49e-49f8-b26e-429659b48ad6-istio-envoy\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:16:04.090492 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:04.090375    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwg2v\" (UniqueName: \"kubernetes.io/projected/c88c84eb-d49e-49f8-b26e-429659b48ad6-kube-api-access-nwg2v\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:16:04.403873 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:04.403834    5061 patch_prober.go:29] interesting pod/e2e-api-a-25de17-openshift-default-8687bbf6cc-p9mf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body=
Oct 30 22:16:04.404051 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:04.403891    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-25de17-openshift-default-8687bbf6cc-p9mf7" podUID="f464e173-8ada-4978-967f-7897d5bc5823" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused"
Oct 30 22:16:04.777667 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:04.777586    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-bpksk" event={"ID":"c88c84eb-d49e-49f8-b26e-429659b48ad6","Type":"ContainerDied","Data":"2a6734158b36a1ecf8da88cb7db5caf2d20c24af9c14631ab97c48c67b898c74"}
Oct 30 22:16:04.777667 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:04.777607    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-bpksk"
Oct 30 22:16:04.777667 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:04.777648    5061 scope.go:122] "RemoveContainer" containerID="e99ff9345b2bd8c8ff797cf09541c95857c43bb9d33cbf9c830df59f0b23e2fc"
Oct 30 22:16:04.991608 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:04.991571    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-bpksk"]
Oct 30 22:16:04.997829 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:04.997803    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-bpksk"]
Oct 30 22:16:05.403803 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:05.403769    5061 patch_prober.go:29] interesting pod/e2e-api-a-25de17-openshift-default-8687bbf6cc-p9mf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body=
Oct 30 22:16:05.403985 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:05.403819    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-25de17-openshift-default-8687bbf6cc-p9mf7" podUID="f464e173-8ada-4978-967f-7897d5bc5823" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused"
Oct 30 22:16:06.404081 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:06.404047    5061 patch_prober.go:29] interesting pod/e2e-api-a-25de17-openshift-default-8687bbf6cc-p9mf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body=
Oct 30 22:16:06.404480 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:06.404096    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-25de17-openshift-default-8687bbf6cc-p9mf7" podUID="f464e173-8ada-4978-967f-7897d5bc5823" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused"
Oct 30 22:16:06.541498 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:06.541464    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c88c84eb-d49e-49f8-b26e-429659b48ad6" path="/var/lib/kubelet/pods/c88c84eb-d49e-49f8-b26e-429659b48ad6/volumes"
Oct 30 22:16:07.408578 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:07.408518    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-25de17-openshift-default-8687bbf6cc-p9mf7"
Oct 30 22:16:07.409043 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:07.409026    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-25de17-openshift-default-8687bbf6cc-p9mf7"
Oct 30 22:16:07.409624 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:07.409607    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-25de17-openshift-default-8687bbf6cc-p9mf7"
Oct 30 22:16:15.799154 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:15.799059    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-25de17-2kqvc"]
Oct 30 22:16:15.799823 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:15.799806    5061 cpu_manager.go:414] "containerMap: removing container" podUID="b0abd91d-7465-4e7f-a840-827ddf311ce7" containerName="payload-processing"
Oct 30 22:16:15.799823 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:15.799823    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0abd91d-7465-4e7f-a840-827ddf311ce7" containerName="payload-processing"
Oct 30 22:16:15.799947 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:15.799836    5061 cpu_manager.go:414] "containerMap: removing container" podUID="081e86b3-a830-4a91-a7da-c270baca6eee" containerName="istio-proxy"
Oct 30 22:16:15.799947 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:15.799842    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="081e86b3-a830-4a91-a7da-c270baca6eee" containerName="istio-proxy"
Oct 30 22:16:15.799947 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:15.799887    5061 cpu_manager.go:414] "containerMap: removing container" podUID="c88c84eb-d49e-49f8-b26e-429659b48ad6" containerName="istio-proxy"
Oct 30 22:16:15.799947 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:15.799893    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c88c84eb-d49e-49f8-b26e-429659b48ad6" containerName="istio-proxy"
Oct 30 22:16:15.799947 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:15.799905    5061 cpu_manager.go:414] "containerMap: removing container" podUID="f9ec2d20-2901-4be4-8b81-0fb125cb4025" containerName="revoke-keys"
Oct 30 22:16:15.799947 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:15.799912    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9ec2d20-2901-4be4-8b81-0fb125cb4025" containerName="revoke-keys"
Oct 30 22:16:15.800201 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:15.800104    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0abd91d-7465-4e7f-a840-827ddf311ce7" containerName="payload-processing"
Oct 30 22:16:15.800201 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:15.800116    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="081e86b3-a830-4a91-a7da-c270baca6eee" containerName="istio-proxy"
Oct 30 22:16:15.800201 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:15.800130    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="c88c84eb-d49e-49f8-b26e-429659b48ad6" containerName="istio-proxy"
Oct 30 22:16:15.800201 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:15.800138    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9ec2d20-2901-4be4-8b81-0fb125cb4025" containerName="revoke-keys"
Oct 30 22:16:15.805411 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:15.805393    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-25de17-2kqvc"
Oct 30 22:16:15.881311 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:15.881275    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8134f480-834a-41ff-ac9a-383a760aecf4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-25de17-2kqvc\" (UID: \"8134f480-834a-41ff-ac9a-383a760aecf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-25de17-2kqvc"
Oct 30 22:16:15.881497 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:15.881387    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8134f480-834a-41ff-ac9a-383a760aecf4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-25de17-2kqvc\" (UID: \"8134f480-834a-41ff-ac9a-383a760aecf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-25de17-2kqvc"
Oct 30 22:16:15.881616 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:15.881593    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lz6h\" (UniqueName: \"kubernetes.io/projected/8134f480-834a-41ff-ac9a-383a760aecf4-kube-api-access-2lz6h\") pod \"maas-api-revoke-keys-e2e-api-a-25de17-2kqvc\" (UID: \"8134f480-834a-41ff-ac9a-383a760aecf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-25de17-2kqvc"
Oct 30 22:16:15.903554 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:15.903514    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:16:15.903705 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:15.903573    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gdmfg\""
Oct 30 22:16:15.907785 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:15.907758    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-25de17-2kqvc"]
Oct 30 22:16:15.983030 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:15.982999    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8134f480-834a-41ff-ac9a-383a760aecf4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-25de17-2kqvc\" (UID: \"8134f480-834a-41ff-ac9a-383a760aecf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-25de17-2kqvc"
Oct 30 22:16:15.983230 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:15.983043    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8134f480-834a-41ff-ac9a-383a760aecf4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-25de17-2kqvc\" (UID: \"8134f480-834a-41ff-ac9a-383a760aecf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-25de17-2kqvc"
Oct 30 22:16:15.983230 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:15.983129    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lz6h\" (UniqueName: \"kubernetes.io/projected/8134f480-834a-41ff-ac9a-383a760aecf4-kube-api-access-2lz6h\") pod \"maas-api-revoke-keys-e2e-api-a-25de17-2kqvc\" (UID: \"8134f480-834a-41ff-ac9a-383a760aecf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-25de17-2kqvc"
Oct 30 22:16:15.983799 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:15.983776    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8134f480-834a-41ff-ac9a-383a760aecf4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-25de17-2kqvc\" (UID: \"8134f480-834a-41ff-ac9a-383a760aecf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-25de17-2kqvc"
Oct 30 22:16:15.985391 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:15.985368    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8134f480-834a-41ff-ac9a-383a760aecf4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-25de17-2kqvc\" (UID: \"8134f480-834a-41ff-ac9a-383a760aecf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-25de17-2kqvc"
Oct 30 22:16:16.037647 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:16.037619    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lz6h\" (UniqueName: \"kubernetes.io/projected/8134f480-834a-41ff-ac9a-383a760aecf4-kube-api-access-2lz6h\") pod \"maas-api-revoke-keys-e2e-api-a-25de17-2kqvc\" (UID: \"8134f480-834a-41ff-ac9a-383a760aecf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-25de17-2kqvc"
Oct 30 22:16:16.121067 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:16.120984    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-25de17-2kqvc"
Oct 30 22:16:16.491447 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:16.491413    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-25de17-2kqvc"]
Oct 30 22:16:16.491715 ip-10-0-130-60 kubenswrapper[5061]: W1030 22:16:16.491576    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8134f480_834a_41ff_ac9a_383a760aecf4.slice/crio-6a65f294f3263abb14b50c2c2a6bbd88965c823d444dc3c55956e25992149c39 WatchSource:0}: Error finding container 6a65f294f3263abb14b50c2c2a6bbd88965c823d444dc3c55956e25992149c39: Status 404 returned error can't find the container with id 6a65f294f3263abb14b50c2c2a6bbd88965c823d444dc3c55956e25992149c39
Oct 30 22:16:16.803830 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:16.803798    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-6ea8da-4phr7"]
Oct 30 22:16:16.809641 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:16.809620    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-6ea8da-4phr7"
Oct 30 22:16:16.866080 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:16.866032    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-6ea8da-4phr7"]
Oct 30 22:16:16.871803 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:16.871768    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-25de17-2kqvc" event={"ID":"8134f480-834a-41ff-ac9a-383a760aecf4","Type":"ContainerStarted","Data":"5571afc0fad36f78a34143e095c058d4b26ec985bdbce5b8136a7e81f50c0f52"}
Oct 30 22:16:16.871932 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:16.871829    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-25de17-2kqvc" event={"ID":"8134f480-834a-41ff-ac9a-383a760aecf4","Type":"ContainerStarted","Data":"6a65f294f3263abb14b50c2c2a6bbd88965c823d444dc3c55956e25992149c39"}
Oct 30 22:16:16.895939 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:16.895900    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44d666c8-a3a5-4560-a446-ca4c8e4ecbca-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-6ea8da-4phr7\" (UID: \"44d666c8-a3a5-4560-a446-ca4c8e4ecbca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-6ea8da-4phr7"
Oct 30 22:16:16.896083 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:16.895972    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crm8z\" (UniqueName: \"kubernetes.io/projected/44d666c8-a3a5-4560-a446-ca4c8e4ecbca-kube-api-access-crm8z\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-6ea8da-4phr7\" (UID: \"44d666c8-a3a5-4560-a446-ca4c8e4ecbca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-6ea8da-4phr7"
Oct 30 22:16:16.896083 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:16.896024    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44d666c8-a3a5-4560-a446-ca4c8e4ecbca-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-6ea8da-4phr7\" (UID: \"44d666c8-a3a5-4560-a446-ca4c8e4ecbca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-6ea8da-4phr7"
Oct 30 22:16:16.997898 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:16.997860    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44d666c8-a3a5-4560-a446-ca4c8e4ecbca-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-6ea8da-4phr7\" (UID: \"44d666c8-a3a5-4560-a446-ca4c8e4ecbca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-6ea8da-4phr7"
Oct 30 22:16:16.998078 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:16.997940    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crm8z\" (UniqueName: \"kubernetes.io/projected/44d666c8-a3a5-4560-a446-ca4c8e4ecbca-kube-api-access-crm8z\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-6ea8da-4phr7\" (UID: \"44d666c8-a3a5-4560-a446-ca4c8e4ecbca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-6ea8da-4phr7"
Oct 30 22:16:16.998078 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:16.997970    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44d666c8-a3a5-4560-a446-ca4c8e4ecbca-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-6ea8da-4phr7\" (UID: \"44d666c8-a3a5-4560-a446-ca4c8e4ecbca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-6ea8da-4phr7"
Oct 30 22:16:16.998710 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:16.998684    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44d666c8-a3a5-4560-a446-ca4c8e4ecbca-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-6ea8da-4phr7\" (UID: \"44d666c8-a3a5-4560-a446-ca4c8e4ecbca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-6ea8da-4phr7"
Oct 30 22:16:17.000423 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:17.000405    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44d666c8-a3a5-4560-a446-ca4c8e4ecbca-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-6ea8da-4phr7\" (UID: \"44d666c8-a3a5-4560-a446-ca4c8e4ecbca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-6ea8da-4phr7"
Oct 30 22:16:17.031927 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:17.031882    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-25de17-2kqvc" podStartSLOduration=2.031870535 podStartE2EDuration="2.031870535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:16:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:16:17.028252025 +0000 UTC m=+2030.641021795" watchObservedRunningTime="2026-10-30 22:16:17.031870535 +0000 UTC m=+2030.644640293"
Oct 30 22:16:17.052858 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:17.052832    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crm8z\" (UniqueName: \"kubernetes.io/projected/44d666c8-a3a5-4560-a446-ca4c8e4ecbca-kube-api-access-crm8z\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-6ea8da-4phr7\" (UID: \"44d666c8-a3a5-4560-a446-ca4c8e4ecbca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-6ea8da-4phr7"
Oct 30 22:16:17.124774 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:17.124711    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-6ea8da-4phr7"
Oct 30 22:16:17.499088 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:17.499056    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-6ea8da-4phr7"]
Oct 30 22:16:17.499362 ip-10-0-130-60 kubenswrapper[5061]: W1030 22:16:17.499340    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44d666c8_a3a5_4560_a446_ca4c8e4ecbca.slice/crio-442092e66fa6500fcff6219a066c0c0eee0209316bb97e9a131e29a338cffeda WatchSource:0}: Error finding container 442092e66fa6500fcff6219a066c0c0eee0209316bb97e9a131e29a338cffeda: Status 404 returned error can't find the container with id 442092e66fa6500fcff6219a066c0c0eee0209316bb97e9a131e29a338cffeda
Oct 30 22:16:17.880496 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:17.880459    5061 generic.go:356] "Generic (PLEG): container finished" podID="8134f480-834a-41ff-ac9a-383a760aecf4" containerID="5571afc0fad36f78a34143e095c058d4b26ec985bdbce5b8136a7e81f50c0f52" exitCode=0
Oct 30 22:16:17.880981 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:17.880559    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-25de17-2kqvc" event={"ID":"8134f480-834a-41ff-ac9a-383a760aecf4","Type":"ContainerDied","Data":"5571afc0fad36f78a34143e095c058d4b26ec985bdbce5b8136a7e81f50c0f52"}
Oct 30 22:16:17.882328 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:17.882304    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-6ea8da-4phr7" event={"ID":"44d666c8-a3a5-4560-a446-ca4c8e4ecbca","Type":"ContainerStarted","Data":"1655dcb183590773dc6a0807ee1b18d3088151835562d856752f3d4644208e06"}
Oct 30 22:16:17.882443 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:17.882357    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-6ea8da-4phr7" event={"ID":"44d666c8-a3a5-4560-a446-ca4c8e4ecbca","Type":"ContainerStarted","Data":"442092e66fa6500fcff6219a066c0c0eee0209316bb97e9a131e29a338cffeda"}
Oct 30 22:16:18.113869 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:18.113747    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-6ea8da-4phr7" podStartSLOduration=2.113733766 podStartE2EDuration="2.113733766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:16:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:16:18.108664458 +0000 UTC m=+2031.721434215" watchObservedRunningTime="2026-10-30 22:16:18.113733766 +0000 UTC m=+2031.726503538"
Oct 30 22:16:18.890961 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:18.890927    5061 generic.go:356] "Generic (PLEG): container finished" podID="44d666c8-a3a5-4560-a446-ca4c8e4ecbca" containerID="1655dcb183590773dc6a0807ee1b18d3088151835562d856752f3d4644208e06" exitCode=0
Oct 30 22:16:18.891411 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:18.891009    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-6ea8da-4phr7" event={"ID":"44d666c8-a3a5-4560-a446-ca4c8e4ecbca","Type":"ContainerDied","Data":"1655dcb183590773dc6a0807ee1b18d3088151835562d856752f3d4644208e06"}
Oct 30 22:16:19.071852 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:19.071827    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-25de17-2kqvc"
Oct 30 22:16:19.129959 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:19.129931    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8134f480-834a-41ff-ac9a-383a760aecf4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8134f480-834a-41ff-ac9a-383a760aecf4-maas-api-openshift-ca\") pod \"8134f480-834a-41ff-ac9a-383a760aecf4\" (UID: \"8134f480-834a-41ff-ac9a-383a760aecf4\") "
Oct 30 22:16:19.130164 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:19.130073    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8134f480-834a-41ff-ac9a-383a760aecf4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8134f480-834a-41ff-ac9a-383a760aecf4-maas-api-xks-ca\") pod \"8134f480-834a-41ff-ac9a-383a760aecf4\" (UID: \"8134f480-834a-41ff-ac9a-383a760aecf4\") "
Oct 30 22:16:19.130164 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:19.130149    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8134f480-834a-41ff-ac9a-383a760aecf4-kube-api-access-2lz6h\" (UniqueName: \"kubernetes.io/projected/8134f480-834a-41ff-ac9a-383a760aecf4-kube-api-access-2lz6h\") pod \"8134f480-834a-41ff-ac9a-383a760aecf4\" (UID: \"8134f480-834a-41ff-ac9a-383a760aecf4\") "
Oct 30 22:16:19.130360 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:19.130322    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8134f480-834a-41ff-ac9a-383a760aecf4-maas-api-openshift-ca" pod "8134f480-834a-41ff-ac9a-383a760aecf4" (UID: "8134f480-834a-41ff-ac9a-383a760aecf4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:16:19.130679 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:19.130651    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8134f480-834a-41ff-ac9a-383a760aecf4-maas-api-openshift-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:16:19.132130 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:19.132105    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8134f480-834a-41ff-ac9a-383a760aecf4-maas-api-xks-ca" pod "8134f480-834a-41ff-ac9a-383a760aecf4" (UID: "8134f480-834a-41ff-ac9a-383a760aecf4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:16:19.132232 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:19.132217    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8134f480-834a-41ff-ac9a-383a760aecf4-kube-api-access-2lz6h" pod "8134f480-834a-41ff-ac9a-383a760aecf4" (UID: "8134f480-834a-41ff-ac9a-383a760aecf4"). InnerVolumeSpecName "kube-api-access-2lz6h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:16:19.232454 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:19.232417    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8134f480-834a-41ff-ac9a-383a760aecf4-maas-api-xks-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:16:19.232454 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:19.232448    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lz6h\" (UniqueName: \"kubernetes.io/projected/8134f480-834a-41ff-ac9a-383a760aecf4-kube-api-access-2lz6h\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:16:19.899495 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:19.899460    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-25de17-2kqvc"
Oct 30 22:16:19.900087 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:19.899460    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-25de17-2kqvc" event={"ID":"8134f480-834a-41ff-ac9a-383a760aecf4","Type":"ContainerDied","Data":"6a65f294f3263abb14b50c2c2a6bbd88965c823d444dc3c55956e25992149c39"}
Oct 30 22:16:19.900087 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:19.899603    5061 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6a65f294f3263abb14b50c2c2a6bbd88965c823d444dc3c55956e25992149c39"
Oct 30 22:16:20.072187 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:20.072158    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-6ea8da-4phr7"
Oct 30 22:16:20.144708 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:20.144679    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44d666c8-a3a5-4560-a446-ca4c8e4ecbca-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44d666c8-a3a5-4560-a446-ca4c8e4ecbca-maas-api-openshift-ca\") pod \"44d666c8-a3a5-4560-a446-ca4c8e4ecbca\" (UID: \"44d666c8-a3a5-4560-a446-ca4c8e4ecbca\") "
Oct 30 22:16:20.144910 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:20.144747    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/44d666c8-a3a5-4560-a446-ca4c8e4ecbca-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44d666c8-a3a5-4560-a446-ca4c8e4ecbca-maas-api-xks-ca\") pod \"44d666c8-a3a5-4560-a446-ca4c8e4ecbca\" (UID: \"44d666c8-a3a5-4560-a446-ca4c8e4ecbca\") "
Oct 30 22:16:20.144910 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:20.144835    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44d666c8-a3a5-4560-a446-ca4c8e4ecbca-kube-api-access-crm8z\" (UniqueName: \"kubernetes.io/projected/44d666c8-a3a5-4560-a446-ca4c8e4ecbca-kube-api-access-crm8z\") pod \"44d666c8-a3a5-4560-a446-ca4c8e4ecbca\" (UID: \"44d666c8-a3a5-4560-a446-ca4c8e4ecbca\") "
Oct 30 22:16:20.145114 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:20.145091    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44d666c8-a3a5-4560-a446-ca4c8e4ecbca-maas-api-openshift-ca" pod "44d666c8-a3a5-4560-a446-ca4c8e4ecbca" (UID: "44d666c8-a3a5-4560-a446-ca4c8e4ecbca"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:16:20.145342 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:20.145325    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44d666c8-a3a5-4560-a446-ca4c8e4ecbca-maas-api-openshift-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:16:20.146909 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:20.146888    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44d666c8-a3a5-4560-a446-ca4c8e4ecbca-maas-api-xks-ca" pod "44d666c8-a3a5-4560-a446-ca4c8e4ecbca" (UID: "44d666c8-a3a5-4560-a446-ca4c8e4ecbca"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:16:20.146999 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:20.146908    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44d666c8-a3a5-4560-a446-ca4c8e4ecbca-kube-api-access-crm8z" pod "44d666c8-a3a5-4560-a446-ca4c8e4ecbca" (UID: "44d666c8-a3a5-4560-a446-ca4c8e4ecbca"). InnerVolumeSpecName "kube-api-access-crm8z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:16:20.247106 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:20.247069    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44d666c8-a3a5-4560-a446-ca4c8e4ecbca-maas-api-xks-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:16:20.247106 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:20.247096    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crm8z\" (UniqueName: \"kubernetes.io/projected/44d666c8-a3a5-4560-a446-ca4c8e4ecbca-kube-api-access-crm8z\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:16:20.907880 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:20.907848    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-6ea8da-4phr7"
Oct 30 22:16:20.908368 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:20.907897    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-6ea8da-4phr7" event={"ID":"44d666c8-a3a5-4560-a446-ca4c8e4ecbca","Type":"ContainerDied","Data":"442092e66fa6500fcff6219a066c0c0eee0209316bb97e9a131e29a338cffeda"}
Oct 30 22:16:20.908368 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:20.907949    5061 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="442092e66fa6500fcff6219a066c0c0eee0209316bb97e9a131e29a338cffeda"
Oct 30 22:16:24.667900 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:16:24.667865    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 22:16:30.595953 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:30.595916    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-25de17-68d48cf4fd-hn6nb"]
Oct 30 22:16:30.696211 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:30.696176    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-25de17-757499fd87-8g2x5"]
Oct 30 22:16:35.202000 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:35.201963    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-446040-hwqrd"]
Oct 30 22:16:35.248494 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:35.248462    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-446040-hwqrd"]
Oct 30 22:16:35.596272 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:35.596215    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-25de17-68d48cf4fd-hn6nb" podUID="b6c1f484-09fa-46b7-a171-d481e895efa9" containerName="payload-processing" containerID="cri-o://749d5687071772c6312c4f7429423ff66e645428ae027df81e2d3a01f92e201b" gracePeriod=25
Oct 30 22:16:35.696604 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:35.696571    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-25de17-757499fd87-8g2x5" podUID="ef968170-02da-4a02-8182-8a5dbb14d62c" containerName="payload-pre-processing" containerID="cri-o://b5157211072ded63f5137411287ab05e92c7098bc4b2eebcec99ee7a47218e75" gracePeriod=25
Oct 30 22:16:35.885271 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:35.884962    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-25de17-68d48cf4fd-hn6nb"
Oct 30 22:16:35.905214 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:35.905178    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6c1f484-09fa-46b7-a171-d481e895efa9-kube-api-access-slhqj\" (UniqueName: \"kubernetes.io/projected/b6c1f484-09fa-46b7-a171-d481e895efa9-kube-api-access-slhqj\") pod \"b6c1f484-09fa-46b7-a171-d481e895efa9\" (UID: \"b6c1f484-09fa-46b7-a171-d481e895efa9\") "
Oct 30 22:16:35.905426 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:35.905240    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6c1f484-09fa-46b7-a171-d481e895efa9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6c1f484-09fa-46b7-a171-d481e895efa9-plugins-config-volume\") pod \"b6c1f484-09fa-46b7-a171-d481e895efa9\" (UID: \"b6c1f484-09fa-46b7-a171-d481e895efa9\") "
Oct 30 22:16:35.905835 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:35.905810    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6c1f484-09fa-46b7-a171-d481e895efa9-plugins-config-volume" pod "b6c1f484-09fa-46b7-a171-d481e895efa9" (UID: "b6c1f484-09fa-46b7-a171-d481e895efa9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:16:35.906012 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:35.905964    5061 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6c1f484-09fa-46b7-a171-d481e895efa9-plugins-config-volume\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:16:35.914168 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:35.908183    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6c1f484-09fa-46b7-a171-d481e895efa9-kube-api-access-slhqj" pod "b6c1f484-09fa-46b7-a171-d481e895efa9" (UID: "b6c1f484-09fa-46b7-a171-d481e895efa9"). InnerVolumeSpecName "kube-api-access-slhqj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:16:35.985075 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:35.985049    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-25de17-757499fd87-8g2x5"
Oct 30 22:16:36.007423 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:36.007390    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef968170-02da-4a02-8182-8a5dbb14d62c-kube-api-access-5j8hs\" (UniqueName: \"kubernetes.io/projected/ef968170-02da-4a02-8182-8a5dbb14d62c-kube-api-access-5j8hs\") pod \"ef968170-02da-4a02-8182-8a5dbb14d62c\" (UID: \"ef968170-02da-4a02-8182-8a5dbb14d62c\") "
Oct 30 22:16:36.007682 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:36.007661    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef968170-02da-4a02-8182-8a5dbb14d62c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef968170-02da-4a02-8182-8a5dbb14d62c-plugins-config-volume\") pod \"ef968170-02da-4a02-8182-8a5dbb14d62c\" (UID: \"ef968170-02da-4a02-8182-8a5dbb14d62c\") "
Oct 30 22:16:36.009688 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:36.008267    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slhqj\" (UniqueName: \"kubernetes.io/projected/b6c1f484-09fa-46b7-a171-d481e895efa9-kube-api-access-slhqj\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:16:36.009688 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:36.008823    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef968170-02da-4a02-8182-8a5dbb14d62c-plugins-config-volume" pod "ef968170-02da-4a02-8182-8a5dbb14d62c" (UID: "ef968170-02da-4a02-8182-8a5dbb14d62c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:16:36.011208 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:36.011187    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef968170-02da-4a02-8182-8a5dbb14d62c-kube-api-access-5j8hs" pod "ef968170-02da-4a02-8182-8a5dbb14d62c" (UID: "ef968170-02da-4a02-8182-8a5dbb14d62c"). InnerVolumeSpecName "kube-api-access-5j8hs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:16:36.092288 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:36.092254    5061 generic.go:356] "Generic (PLEG): container finished" podID="ef968170-02da-4a02-8182-8a5dbb14d62c" containerID="b5157211072ded63f5137411287ab05e92c7098bc4b2eebcec99ee7a47218e75" exitCode=0
Oct 30 22:16:36.092466 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:36.092299    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-25de17-757499fd87-8g2x5"
Oct 30 22:16:36.092466 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:36.092336    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-25de17-757499fd87-8g2x5" event={"ID":"ef968170-02da-4a02-8182-8a5dbb14d62c","Type":"ContainerDied","Data":"b5157211072ded63f5137411287ab05e92c7098bc4b2eebcec99ee7a47218e75"}
Oct 30 22:16:36.092466 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:36.092389    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-25de17-757499fd87-8g2x5" event={"ID":"ef968170-02da-4a02-8182-8a5dbb14d62c","Type":"ContainerDied","Data":"d8593939c92433cd2a02894644b808a1c66b487423b77d84ffa7f36e65ed6f31"}
Oct 30 22:16:36.092466 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:36.092414    5061 scope.go:122] "RemoveContainer" containerID="b5157211072ded63f5137411287ab05e92c7098bc4b2eebcec99ee7a47218e75"
Oct 30 22:16:36.093890 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:36.093863    5061 generic.go:356] "Generic (PLEG): container finished" podID="b6c1f484-09fa-46b7-a171-d481e895efa9" containerID="749d5687071772c6312c4f7429423ff66e645428ae027df81e2d3a01f92e201b" exitCode=0
Oct 30 22:16:36.094030 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:36.093909    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-25de17-68d48cf4fd-hn6nb"
Oct 30 22:16:36.094030 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:36.093982    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-25de17-68d48cf4fd-hn6nb" event={"ID":"b6c1f484-09fa-46b7-a171-d481e895efa9","Type":"ContainerDied","Data":"749d5687071772c6312c4f7429423ff66e645428ae027df81e2d3a01f92e201b"}
Oct 30 22:16:36.094162 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:36.094044    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-25de17-68d48cf4fd-hn6nb" event={"ID":"b6c1f484-09fa-46b7-a171-d481e895efa9","Type":"ContainerDied","Data":"32d535998ba22477462a966dd2723653a7a0747c38dd94cc1bf1253b00daa5be"}
Oct 30 22:16:36.103217 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:36.103199    5061 scope.go:122] "RemoveContainer" containerID="b5157211072ded63f5137411287ab05e92c7098bc4b2eebcec99ee7a47218e75"
Oct 30 22:16:36.103514 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:16:36.103475    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5157211072ded63f5137411287ab05e92c7098bc4b2eebcec99ee7a47218e75\": container with ID starting with b5157211072ded63f5137411287ab05e92c7098bc4b2eebcec99ee7a47218e75 not found: ID does not exist" containerID="b5157211072ded63f5137411287ab05e92c7098bc4b2eebcec99ee7a47218e75"
Oct 30 22:16:36.103652 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:36.103515    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5157211072ded63f5137411287ab05e92c7098bc4b2eebcec99ee7a47218e75"} err="failed to get container status \"b5157211072ded63f5137411287ab05e92c7098bc4b2eebcec99ee7a47218e75\": rpc error: code = NotFound desc = could not find container \"b5157211072ded63f5137411287ab05e92c7098bc4b2eebcec99ee7a47218e75\": container with ID starting with b5157211072ded63f5137411287ab05e92c7098bc4b2eebcec99ee7a47218e75 not found: ID does not exist"
Oct 30 22:16:36.103652 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:36.103556    5061 scope.go:122] "RemoveContainer" containerID="749d5687071772c6312c4f7429423ff66e645428ae027df81e2d3a01f92e201b"
Oct 30 22:16:36.109258 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:36.109235    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5j8hs\" (UniqueName: \"kubernetes.io/projected/ef968170-02da-4a02-8182-8a5dbb14d62c-kube-api-access-5j8hs\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:16:36.109258 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:36.109256    5061 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef968170-02da-4a02-8182-8a5dbb14d62c-plugins-config-volume\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:16:36.113304 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:36.113285    5061 scope.go:122] "RemoveContainer" containerID="749d5687071772c6312c4f7429423ff66e645428ae027df81e2d3a01f92e201b"
Oct 30 22:16:36.113588 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:16:36.113568    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"749d5687071772c6312c4f7429423ff66e645428ae027df81e2d3a01f92e201b\": container with ID starting with 749d5687071772c6312c4f7429423ff66e645428ae027df81e2d3a01f92e201b not found: ID does not exist" containerID="749d5687071772c6312c4f7429423ff66e645428ae027df81e2d3a01f92e201b"
Oct 30 22:16:36.113660 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:36.113599    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"749d5687071772c6312c4f7429423ff66e645428ae027df81e2d3a01f92e201b"} err="failed to get container status \"749d5687071772c6312c4f7429423ff66e645428ae027df81e2d3a01f92e201b\": rpc error: code = NotFound desc = could not find container \"749d5687071772c6312c4f7429423ff66e645428ae027df81e2d3a01f92e201b\": container with ID starting with 749d5687071772c6312c4f7429423ff66e645428ae027df81e2d3a01f92e201b not found: ID does not exist"
Oct 30 22:16:36.222848 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:36.222812    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-25de17-68d48cf4fd-hn6nb"]
Oct 30 22:16:36.227241 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:36.227213    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-25de17-68d48cf4fd-hn6nb"]
Oct 30 22:16:36.331157 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:36.331122    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-25de17-757499fd87-8g2x5"]
Oct 30 22:16:36.335145 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:36.335117    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-25de17-757499fd87-8g2x5"]
Oct 30 22:16:36.530117 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:36.530044    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6c1f484-09fa-46b7-a171-d481e895efa9" path="/var/lib/kubelet/pods/b6c1f484-09fa-46b7-a171-d481e895efa9/volumes"
Oct 30 22:16:36.530472 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:36.530458    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef968170-02da-4a02-8182-8a5dbb14d62c" path="/var/lib/kubelet/pods/ef968170-02da-4a02-8182-8a5dbb14d62c/volumes"
Oct 30 22:16:36.530889 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:36.530876    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9ec2d20-2901-4be4-8b81-0fb125cb4025" path="/var/lib/kubelet/pods/f9ec2d20-2901-4be4-8b81-0fb125cb4025/volumes"
Oct 30 22:16:36.893227 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:36.893185    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-api-a-25de17-757499fd87-8g2x5" podUID="ef968170-02da-4a02-8182-8a5dbb14d62c" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.131:50052\" within 1s: context deadline exceeded"
Oct 30 22:16:36.893406 ip-10-0-130-60 kubenswrapper[5061]: W1030 22:16:36.893242    5061 logging.go:55] [core] [Channel #3899 SubChannel #3900]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.131:50052", ServerName: "10.132.0.131:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.131:50052: operation was canceled"
Oct 30 22:16:38.537207 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:38.537175    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f24db166-hnbss"]
Oct 30 22:16:38.538158 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:38.538133    5061 cpu_manager.go:414] "containerMap: removing container" podUID="44d666c8-a3a5-4560-a446-ca4c8e4ecbca" containerName="revoke-keys"
Oct 30 22:16:38.538289 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:38.538161    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44d666c8-a3a5-4560-a446-ca4c8e4ecbca" containerName="revoke-keys"
Oct 30 22:16:38.538289 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:38.538193    5061 cpu_manager.go:414] "containerMap: removing container" podUID="ef968170-02da-4a02-8182-8a5dbb14d62c" containerName="payload-pre-processing"
Oct 30 22:16:38.538289 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:38.538201    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef968170-02da-4a02-8182-8a5dbb14d62c" containerName="payload-pre-processing"
Oct 30 22:16:38.538289 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:38.538231    5061 cpu_manager.go:414] "containerMap: removing container" podUID="8134f480-834a-41ff-ac9a-383a760aecf4" containerName="revoke-keys"
Oct 30 22:16:38.538289 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:38.538238    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8134f480-834a-41ff-ac9a-383a760aecf4" containerName="revoke-keys"
Oct 30 22:16:38.538552 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:38.538296    5061 cpu_manager.go:414] "containerMap: removing container" podUID="b6c1f484-09fa-46b7-a171-d481e895efa9" containerName="payload-processing"
Oct 30 22:16:38.538552 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:38.538304    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6c1f484-09fa-46b7-a171-d481e895efa9" containerName="payload-processing"
Oct 30 22:16:38.538552 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:38.538473    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="8134f480-834a-41ff-ac9a-383a760aecf4" containerName="revoke-keys"
Oct 30 22:16:38.538552 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:38.538496    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6c1f484-09fa-46b7-a171-d481e895efa9" containerName="payload-processing"
Oct 30 22:16:38.538552 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:38.538505    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="44d666c8-a3a5-4560-a446-ca4c8e4ecbca" containerName="revoke-keys"
Oct 30 22:16:38.538552 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:38.538513    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef968170-02da-4a02-8182-8a5dbb14d62c" containerName="payload-pre-processing"
Oct 30 22:16:38.543669 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:38.543652    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f24db166-hnbss"
Oct 30 22:16:38.599275 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:38.599241    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f24db166-hnbss"]
Oct 30 22:16:38.642700 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:38.642665    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:16:38.642956 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:38.642712    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gdmfg\""
Oct 30 22:16:38.643240 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:38.643212    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d2a7ba6-ea4c-4425-88c2-415faa0b0db5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f24db166-hnbss\" (UID: \"7d2a7ba6-ea4c-4425-88c2-415faa0b0db5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f24db166-hnbss"
Oct 30 22:16:38.643381 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:38.643360    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2c5t2\" (UniqueName: \"kubernetes.io/projected/7d2a7ba6-ea4c-4425-88c2-415faa0b0db5-kube-api-access-2c5t2\") pod \"maas-api-revoke-sub-f24db166-hnbss\" (UID: \"7d2a7ba6-ea4c-4425-88c2-415faa0b0db5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f24db166-hnbss"
Oct 30 22:16:38.643467 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:38.643437    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d2a7ba6-ea4c-4425-88c2-415faa0b0db5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f24db166-hnbss\" (UID: \"7d2a7ba6-ea4c-4425-88c2-415faa0b0db5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f24db166-hnbss"
Oct 30 22:16:38.745442 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:38.745409    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d2a7ba6-ea4c-4425-88c2-415faa0b0db5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f24db166-hnbss\" (UID: \"7d2a7ba6-ea4c-4425-88c2-415faa0b0db5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f24db166-hnbss"
Oct 30 22:16:38.745607 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:38.745468    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2c5t2\" (UniqueName: \"kubernetes.io/projected/7d2a7ba6-ea4c-4425-88c2-415faa0b0db5-kube-api-access-2c5t2\") pod \"maas-api-revoke-sub-f24db166-hnbss\" (UID: \"7d2a7ba6-ea4c-4425-88c2-415faa0b0db5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f24db166-hnbss"
Oct 30 22:16:38.745607 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:38.745501    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d2a7ba6-ea4c-4425-88c2-415faa0b0db5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f24db166-hnbss\" (UID: \"7d2a7ba6-ea4c-4425-88c2-415faa0b0db5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f24db166-hnbss"
Oct 30 22:16:38.746123 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:38.746103    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d2a7ba6-ea4c-4425-88c2-415faa0b0db5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f24db166-hnbss\" (UID: \"7d2a7ba6-ea4c-4425-88c2-415faa0b0db5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f24db166-hnbss"
Oct 30 22:16:38.747865 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:38.747844    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d2a7ba6-ea4c-4425-88c2-415faa0b0db5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f24db166-hnbss\" (UID: \"7d2a7ba6-ea4c-4425-88c2-415faa0b0db5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f24db166-hnbss"
Oct 30 22:16:38.799649 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:38.799624    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2c5t2\" (UniqueName: \"kubernetes.io/projected/7d2a7ba6-ea4c-4425-88c2-415faa0b0db5-kube-api-access-2c5t2\") pod \"maas-api-revoke-sub-f24db166-hnbss\" (UID: \"7d2a7ba6-ea4c-4425-88c2-415faa0b0db5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f24db166-hnbss"
Oct 30 22:16:38.857785 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:38.857759    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f24db166-hnbss"
Oct 30 22:16:39.238332 ip-10-0-130-60 kubenswrapper[5061]: W1030 22:16:39.238296    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d2a7ba6_ea4c_4425_88c2_415faa0b0db5.slice/crio-185665dd812b6d5f6e2271d75aae324b1f4d5dbae72e6d86eb4808e5172011c8 WatchSource:0}: Error finding container 185665dd812b6d5f6e2271d75aae324b1f4d5dbae72e6d86eb4808e5172011c8: Status 404 returned error can't find the container with id 185665dd812b6d5f6e2271d75aae324b1f4d5dbae72e6d86eb4808e5172011c8
Oct 30 22:16:39.248804 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:39.247792    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f24db166-hnbss"]
Oct 30 22:16:40.128130 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:40.128103    5061 generic.go:356] "Generic (PLEG): container finished" podID="7d2a7ba6-ea4c-4425-88c2-415faa0b0db5" containerID="c563dc078d52100e9275c06e500dc0100c4e027130e1ed1a1b8474455495e381" exitCode=0
Oct 30 22:16:40.128491 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:40.128201    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f24db166-hnbss" event={"ID":"7d2a7ba6-ea4c-4425-88c2-415faa0b0db5","Type":"ContainerDied","Data":"c563dc078d52100e9275c06e500dc0100c4e027130e1ed1a1b8474455495e381"}
Oct 30 22:16:40.128491 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:40.128274    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f24db166-hnbss" event={"ID":"7d2a7ba6-ea4c-4425-88c2-415faa0b0db5","Type":"ContainerStarted","Data":"185665dd812b6d5f6e2271d75aae324b1f4d5dbae72e6d86eb4808e5172011c8"}
Oct 30 22:16:41.301575 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:41.301549    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f24db166-hnbss"
Oct 30 22:16:41.383273 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:41.383242    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d2a7ba6-ea4c-4425-88c2-415faa0b0db5-kube-api-access-2c5t2\" (UniqueName: \"kubernetes.io/projected/7d2a7ba6-ea4c-4425-88c2-415faa0b0db5-kube-api-access-2c5t2\") pod \"7d2a7ba6-ea4c-4425-88c2-415faa0b0db5\" (UID: \"7d2a7ba6-ea4c-4425-88c2-415faa0b0db5\") "
Oct 30 22:16:41.383431 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:41.383344    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d2a7ba6-ea4c-4425-88c2-415faa0b0db5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d2a7ba6-ea4c-4425-88c2-415faa0b0db5-maas-api-xks-ca\") pod \"7d2a7ba6-ea4c-4425-88c2-415faa0b0db5\" (UID: \"7d2a7ba6-ea4c-4425-88c2-415faa0b0db5\") "
Oct 30 22:16:41.383431 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:41.383378    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d2a7ba6-ea4c-4425-88c2-415faa0b0db5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d2a7ba6-ea4c-4425-88c2-415faa0b0db5-maas-api-openshift-ca\") pod \"7d2a7ba6-ea4c-4425-88c2-415faa0b0db5\" (UID: \"7d2a7ba6-ea4c-4425-88c2-415faa0b0db5\") "
Oct 30 22:16:41.383797 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:41.383768    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d2a7ba6-ea4c-4425-88c2-415faa0b0db5-maas-api-openshift-ca" pod "7d2a7ba6-ea4c-4425-88c2-415faa0b0db5" (UID: "7d2a7ba6-ea4c-4425-88c2-415faa0b0db5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:16:41.384093 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:41.384074    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d2a7ba6-ea4c-4425-88c2-415faa0b0db5-maas-api-openshift-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:16:41.385596 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:41.385514    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d2a7ba6-ea4c-4425-88c2-415faa0b0db5-maas-api-xks-ca" pod "7d2a7ba6-ea4c-4425-88c2-415faa0b0db5" (UID: "7d2a7ba6-ea4c-4425-88c2-415faa0b0db5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:16:41.385596 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:41.385565    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d2a7ba6-ea4c-4425-88c2-415faa0b0db5-kube-api-access-2c5t2" pod "7d2a7ba6-ea4c-4425-88c2-415faa0b0db5" (UID: "7d2a7ba6-ea4c-4425-88c2-415faa0b0db5"). InnerVolumeSpecName "kube-api-access-2c5t2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:16:41.485255 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:41.485220    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2c5t2\" (UniqueName: \"kubernetes.io/projected/7d2a7ba6-ea4c-4425-88c2-415faa0b0db5-kube-api-access-2c5t2\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:16:41.485255 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:41.485251    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d2a7ba6-ea4c-4425-88c2-415faa0b0db5-maas-api-xks-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:16:42.143430 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:42.143396    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f24db166-hnbss"
Oct 30 22:16:42.143632 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:42.143427    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f24db166-hnbss" event={"ID":"7d2a7ba6-ea4c-4425-88c2-415faa0b0db5","Type":"ContainerDied","Data":"185665dd812b6d5f6e2271d75aae324b1f4d5dbae72e6d86eb4808e5172011c8"}
Oct 30 22:16:42.143632 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:42.143478    5061 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="185665dd812b6d5f6e2271d75aae324b1f4d5dbae72e6d86eb4808e5172011c8"
Oct 30 22:16:54.114622 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:54.114578    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-411ff1-lgs7j"]
Oct 30 22:16:54.115518 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:54.115495    5061 cpu_manager.go:414] "containerMap: removing container" podUID="7d2a7ba6-ea4c-4425-88c2-415faa0b0db5" containerName="revoke-keys"
Oct 30 22:16:54.115673 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:54.115519    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d2a7ba6-ea4c-4425-88c2-415faa0b0db5" containerName="revoke-keys"
Oct 30 22:16:54.115736 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:54.115704    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d2a7ba6-ea4c-4425-88c2-415faa0b0db5" containerName="revoke-keys"
Oct 30 22:16:54.119453 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:54.119432    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-411ff1-lgs7j"
Oct 30 22:16:54.171499 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:54.171437    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:16:54.218368 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:54.218328    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gdmfg\""
Oct 30 22:16:54.222107 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:54.222084    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-411ff1-lgs7j"]
Oct 30 22:16:54.253611 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:54.253581    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84c958fa-2860-4399-918e-cbd96d83ccfe-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-411ff1-lgs7j\" (UID: \"84c958fa-2860-4399-918e-cbd96d83ccfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-411ff1-lgs7j"
Oct 30 22:16:54.254172 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:54.253778    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjkvw\" (UniqueName: \"kubernetes.io/projected/84c958fa-2860-4399-918e-cbd96d83ccfe-kube-api-access-xjkvw\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-411ff1-lgs7j\" (UID: \"84c958fa-2860-4399-918e-cbd96d83ccfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-411ff1-lgs7j"
Oct 30 22:16:54.254172 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:54.253805    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84c958fa-2860-4399-918e-cbd96d83ccfe-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-411ff1-lgs7j\" (UID: \"84c958fa-2860-4399-918e-cbd96d83ccfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-411ff1-lgs7j"
Oct 30 22:16:54.355499 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:54.355468    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjkvw\" (UniqueName: \"kubernetes.io/projected/84c958fa-2860-4399-918e-cbd96d83ccfe-kube-api-access-xjkvw\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-411ff1-lgs7j\" (UID: \"84c958fa-2860-4399-918e-cbd96d83ccfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-411ff1-lgs7j"
Oct 30 22:16:54.355685 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:54.355521    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84c958fa-2860-4399-918e-cbd96d83ccfe-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-411ff1-lgs7j\" (UID: \"84c958fa-2860-4399-918e-cbd96d83ccfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-411ff1-lgs7j"
Oct 30 22:16:54.355685 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:54.355674    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84c958fa-2860-4399-918e-cbd96d83ccfe-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-411ff1-lgs7j\" (UID: \"84c958fa-2860-4399-918e-cbd96d83ccfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-411ff1-lgs7j"
Oct 30 22:16:54.356289 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:54.356266    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84c958fa-2860-4399-918e-cbd96d83ccfe-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-411ff1-lgs7j\" (UID: \"84c958fa-2860-4399-918e-cbd96d83ccfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-411ff1-lgs7j"
Oct 30 22:16:54.357869 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:54.357848    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84c958fa-2860-4399-918e-cbd96d83ccfe-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-411ff1-lgs7j\" (UID: \"84c958fa-2860-4399-918e-cbd96d83ccfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-411ff1-lgs7j"
Oct 30 22:16:54.409910 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:54.409852    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjkvw\" (UniqueName: \"kubernetes.io/projected/84c958fa-2860-4399-918e-cbd96d83ccfe-kube-api-access-xjkvw\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-411ff1-lgs7j\" (UID: \"84c958fa-2860-4399-918e-cbd96d83ccfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-411ff1-lgs7j"
Oct 30 22:16:54.436700 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:54.436677    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-411ff1-lgs7j"
Oct 30 22:16:54.667683 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:16:54.667601    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 22:16:54.816154 ip-10-0-130-60 kubenswrapper[5061]: W1030 22:16:54.816113    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84c958fa_2860_4399_918e_cbd96d83ccfe.slice/crio-9e9301abfcebeacc6c977733445c032b7b62bef6f002afd78d9b562e77893539 WatchSource:0}: Error finding container 9e9301abfcebeacc6c977733445c032b7b62bef6f002afd78d9b562e77893539: Status 404 returned error can't find the container with id 9e9301abfcebeacc6c977733445c032b7b62bef6f002afd78d9b562e77893539
Oct 30 22:16:54.817872 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:54.817840    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-411ff1-lgs7j"]
Oct 30 22:16:55.275599 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:55.275481    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-411ff1-lgs7j" event={"ID":"84c958fa-2860-4399-918e-cbd96d83ccfe","Type":"ContainerStarted","Data":"9ead5d84a669e03c1419b5a4f714ee316538a3993a511b0139bfc3f412726020"}
Oct 30 22:16:55.276054 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:55.275619    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-411ff1-lgs7j" event={"ID":"84c958fa-2860-4399-918e-cbd96d83ccfe","Type":"ContainerStarted","Data":"9e9301abfcebeacc6c977733445c032b7b62bef6f002afd78d9b562e77893539"}
Oct 30 22:16:55.386450 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:55.386393    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-411ff1-lgs7j" podStartSLOduration=1.38637838 podStartE2EDuration="1.38637838s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:16:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:16:55.384443595 +0000 UTC m=+2068.997213364" watchObservedRunningTime="2026-10-30 22:16:55.38637838 +0000 UTC m=+2068.999148139"
Oct 30 22:16:56.283848 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:56.283818    5061 generic.go:356] "Generic (PLEG): container finished" podID="84c958fa-2860-4399-918e-cbd96d83ccfe" containerID="9ead5d84a669e03c1419b5a4f714ee316538a3993a511b0139bfc3f412726020" exitCode=0
Oct 30 22:16:56.284242 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:56.283905    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-411ff1-lgs7j" event={"ID":"84c958fa-2860-4399-918e-cbd96d83ccfe","Type":"ContainerDied","Data":"9ead5d84a669e03c1419b5a4f714ee316538a3993a511b0139bfc3f412726020"}
Oct 30 22:16:57.452126 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:57.452100    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-411ff1-lgs7j"
Oct 30 22:16:57.599317 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:57.599284    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84c958fa-2860-4399-918e-cbd96d83ccfe-kube-api-access-xjkvw\" (UniqueName: \"kubernetes.io/projected/84c958fa-2860-4399-918e-cbd96d83ccfe-kube-api-access-xjkvw\") pod \"84c958fa-2860-4399-918e-cbd96d83ccfe\" (UID: \"84c958fa-2860-4399-918e-cbd96d83ccfe\") "
Oct 30 22:16:57.599510 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:57.599338    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84c958fa-2860-4399-918e-cbd96d83ccfe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84c958fa-2860-4399-918e-cbd96d83ccfe-maas-api-openshift-ca\") pod \"84c958fa-2860-4399-918e-cbd96d83ccfe\" (UID: \"84c958fa-2860-4399-918e-cbd96d83ccfe\") "
Oct 30 22:16:57.599510 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:57.599436    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84c958fa-2860-4399-918e-cbd96d83ccfe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84c958fa-2860-4399-918e-cbd96d83ccfe-maas-api-xks-ca\") pod \"84c958fa-2860-4399-918e-cbd96d83ccfe\" (UID: \"84c958fa-2860-4399-918e-cbd96d83ccfe\") "
Oct 30 22:16:57.599799 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:57.599775    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84c958fa-2860-4399-918e-cbd96d83ccfe-maas-api-openshift-ca" pod "84c958fa-2860-4399-918e-cbd96d83ccfe" (UID: "84c958fa-2860-4399-918e-cbd96d83ccfe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:16:57.600566 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:57.600520    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84c958fa-2860-4399-918e-cbd96d83ccfe-maas-api-openshift-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:16:57.602158 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:57.602129    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84c958fa-2860-4399-918e-cbd96d83ccfe-kube-api-access-xjkvw" pod "84c958fa-2860-4399-918e-cbd96d83ccfe" (UID: "84c958fa-2860-4399-918e-cbd96d83ccfe"). InnerVolumeSpecName "kube-api-access-xjkvw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:16:57.602271 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:57.602150    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84c958fa-2860-4399-918e-cbd96d83ccfe-maas-api-xks-ca" pod "84c958fa-2860-4399-918e-cbd96d83ccfe" (UID: "84c958fa-2860-4399-918e-cbd96d83ccfe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:16:57.701847 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:57.701814    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84c958fa-2860-4399-918e-cbd96d83ccfe-maas-api-xks-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:16:57.701847 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:57.701839    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjkvw\" (UniqueName: \"kubernetes.io/projected/84c958fa-2860-4399-918e-cbd96d83ccfe-kube-api-access-xjkvw\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:16:58.298089 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:58.298052    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-411ff1-lgs7j" event={"ID":"84c958fa-2860-4399-918e-cbd96d83ccfe","Type":"ContainerDied","Data":"9e9301abfcebeacc6c977733445c032b7b62bef6f002afd78d9b562e77893539"}
Oct 30 22:16:58.298257 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:58.298102    5061 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9e9301abfcebeacc6c977733445c032b7b62bef6f002afd78d9b562e77893539"
Oct 30 22:16:58.298257 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:16:58.298146    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-411ff1-lgs7j"
Oct 30 22:17:02.417200 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:02.417159    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-263513-hc7m9"]
Oct 30 22:17:02.418076 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:02.418054    5061 cpu_manager.go:414] "containerMap: removing container" podUID="84c958fa-2860-4399-918e-cbd96d83ccfe" containerName="revoke-keys"
Oct 30 22:17:02.418076 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:02.418074    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84c958fa-2860-4399-918e-cbd96d83ccfe" containerName="revoke-keys"
Oct 30 22:17:02.418307 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:02.418219    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="84c958fa-2860-4399-918e-cbd96d83ccfe" containerName="revoke-keys"
Oct 30 22:17:02.422073 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:02.422057    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-263513-hc7m9"
Oct 30 22:17:02.465200 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:02.465171    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4dfd4fc5-bd34-4451-9544-e3357405b66e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-263513-hc7m9\" (UID: \"4dfd4fc5-bd34-4451-9544-e3357405b66e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-263513-hc7m9"
Oct 30 22:17:02.465400 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:02.465377    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4dfd4fc5-bd34-4451-9544-e3357405b66e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-263513-hc7m9\" (UID: \"4dfd4fc5-bd34-4451-9544-e3357405b66e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-263513-hc7m9"
Oct 30 22:17:02.465451 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:02.465416    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhh2p\" (UniqueName: \"kubernetes.io/projected/4dfd4fc5-bd34-4451-9544-e3357405b66e-kube-api-access-nhh2p\") pod \"maas-api-revoke-keys-e2e-inf-a-263513-hc7m9\" (UID: \"4dfd4fc5-bd34-4451-9544-e3357405b66e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-263513-hc7m9"
Oct 30 22:17:02.471635 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:02.471615    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:17:02.518716 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:02.518668    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gdmfg\""
Oct 30 22:17:02.529598 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:02.529572    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-263513-hc7m9"]
Oct 30 22:17:02.566911 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:02.566886    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4dfd4fc5-bd34-4451-9544-e3357405b66e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-263513-hc7m9\" (UID: \"4dfd4fc5-bd34-4451-9544-e3357405b66e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-263513-hc7m9"
Oct 30 22:17:02.567068 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:02.566928    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhh2p\" (UniqueName: \"kubernetes.io/projected/4dfd4fc5-bd34-4451-9544-e3357405b66e-kube-api-access-nhh2p\") pod \"maas-api-revoke-keys-e2e-inf-a-263513-hc7m9\" (UID: \"4dfd4fc5-bd34-4451-9544-e3357405b66e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-263513-hc7m9"
Oct 30 22:17:02.567355 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:02.567321    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4dfd4fc5-bd34-4451-9544-e3357405b66e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-263513-hc7m9\" (UID: \"4dfd4fc5-bd34-4451-9544-e3357405b66e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-263513-hc7m9"
Oct 30 22:17:02.568001 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:02.567978    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4dfd4fc5-bd34-4451-9544-e3357405b66e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-263513-hc7m9\" (UID: \"4dfd4fc5-bd34-4451-9544-e3357405b66e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-263513-hc7m9"
Oct 30 22:17:02.569474 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:02.569450    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4dfd4fc5-bd34-4451-9544-e3357405b66e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-263513-hc7m9\" (UID: \"4dfd4fc5-bd34-4451-9544-e3357405b66e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-263513-hc7m9"
Oct 30 22:17:02.621331 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:02.621306    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhh2p\" (UniqueName: \"kubernetes.io/projected/4dfd4fc5-bd34-4451-9544-e3357405b66e-kube-api-access-nhh2p\") pod \"maas-api-revoke-keys-e2e-inf-a-263513-hc7m9\" (UID: \"4dfd4fc5-bd34-4451-9544-e3357405b66e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-263513-hc7m9"
Oct 30 22:17:02.735915 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:02.735843    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-263513-hc7m9"
Oct 30 22:17:03.184094 ip-10-0-130-60 kubenswrapper[5061]: W1030 22:17:03.184009    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dfd4fc5_bd34_4451_9544_e3357405b66e.slice/crio-1248dfc1385931312e9f82baa8b9f8795e1cef7efc0a43aff0f58bf8fde59e9c WatchSource:0}: Error finding container 1248dfc1385931312e9f82baa8b9f8795e1cef7efc0a43aff0f58bf8fde59e9c: Status 404 returned error can't find the container with id 1248dfc1385931312e9f82baa8b9f8795e1cef7efc0a43aff0f58bf8fde59e9c
Oct 30 22:17:03.184094 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:03.184062    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-263513-hc7m9"]
Oct 30 22:17:03.341249 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:03.341210    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-263513-hc7m9" event={"ID":"4dfd4fc5-bd34-4451-9544-e3357405b66e","Type":"ContainerStarted","Data":"1248dfc1385931312e9f82baa8b9f8795e1cef7efc0a43aff0f58bf8fde59e9c"}
Oct 30 22:17:04.350558 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:04.350499    5061 generic.go:356] "Generic (PLEG): container finished" podID="4dfd4fc5-bd34-4451-9544-e3357405b66e" containerID="beb09a430bb255c20d2be5aeadc0dd3ef8c5740ab0ad00ae7d56263b90d24f06" exitCode=0
Oct 30 22:17:04.350947 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:04.350570    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-263513-hc7m9" event={"ID":"4dfd4fc5-bd34-4451-9544-e3357405b66e","Type":"ContainerDied","Data":"beb09a430bb255c20d2be5aeadc0dd3ef8c5740ab0ad00ae7d56263b90d24f06"}
Oct 30 22:17:05.528965 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:05.528937    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-263513-hc7m9"
Oct 30 22:17:05.609708 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:05.609628    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4dfd4fc5-bd34-4451-9544-e3357405b66e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4dfd4fc5-bd34-4451-9544-e3357405b66e-maas-api-xks-ca\") pod \"4dfd4fc5-bd34-4451-9544-e3357405b66e\" (UID: \"4dfd4fc5-bd34-4451-9544-e3357405b66e\") "
Oct 30 22:17:05.609708 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:05.609668    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4dfd4fc5-bd34-4451-9544-e3357405b66e-kube-api-access-nhh2p\" (UniqueName: \"kubernetes.io/projected/4dfd4fc5-bd34-4451-9544-e3357405b66e-kube-api-access-nhh2p\") pod \"4dfd4fc5-bd34-4451-9544-e3357405b66e\" (UID: \"4dfd4fc5-bd34-4451-9544-e3357405b66e\") "
Oct 30 22:17:05.609920 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:05.609816    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4dfd4fc5-bd34-4451-9544-e3357405b66e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4dfd4fc5-bd34-4451-9544-e3357405b66e-maas-api-openshift-ca\") pod \"4dfd4fc5-bd34-4451-9544-e3357405b66e\" (UID: \"4dfd4fc5-bd34-4451-9544-e3357405b66e\") "
Oct 30 22:17:05.610176 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:05.610148    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4dfd4fc5-bd34-4451-9544-e3357405b66e-maas-api-openshift-ca" pod "4dfd4fc5-bd34-4451-9544-e3357405b66e" (UID: "4dfd4fc5-bd34-4451-9544-e3357405b66e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:17:05.610378 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:05.610362    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4dfd4fc5-bd34-4451-9544-e3357405b66e-maas-api-openshift-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:05.611893 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:05.611871    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4dfd4fc5-bd34-4451-9544-e3357405b66e-maas-api-xks-ca" pod "4dfd4fc5-bd34-4451-9544-e3357405b66e" (UID: "4dfd4fc5-bd34-4451-9544-e3357405b66e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:17:05.612026 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:05.611890    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dfd4fc5-bd34-4451-9544-e3357405b66e-kube-api-access-nhh2p" pod "4dfd4fc5-bd34-4451-9544-e3357405b66e" (UID: "4dfd4fc5-bd34-4451-9544-e3357405b66e"). InnerVolumeSpecName "kube-api-access-nhh2p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:17:05.711316 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:05.711280    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4dfd4fc5-bd34-4451-9544-e3357405b66e-maas-api-xks-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:05.711316 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:05.711314    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhh2p\" (UniqueName: \"kubernetes.io/projected/4dfd4fc5-bd34-4451-9544-e3357405b66e-kube-api-access-nhh2p\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:06.367800 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:06.367769    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-263513-hc7m9"
Oct 30 22:17:06.367993 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:06.367803    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-263513-hc7m9" event={"ID":"4dfd4fc5-bd34-4451-9544-e3357405b66e","Type":"ContainerDied","Data":"1248dfc1385931312e9f82baa8b9f8795e1cef7efc0a43aff0f58bf8fde59e9c"}
Oct 30 22:17:06.367993 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:06.367856    5061 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1248dfc1385931312e9f82baa8b9f8795e1cef7efc0a43aff0f58bf8fde59e9c"
Oct 30 22:17:07.627904 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:07.627866    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-411ff1-794dbcbd69-sfhsl"]
Oct 30 22:17:07.628398 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:07.628073    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-411ff1-794dbcbd69-sfhsl" podUID="2c04bb48-1286-48fe-af0d-fc9ec478e663" containerName="maas-api" containerID="cri-o://4ec166df46a0650130217fff82aeceee87af729bcce56d41cc3dad535784fbd4" gracePeriod=30
Oct 30 22:17:07.981888 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:07.981855    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-411ff1-794dbcbd69-sfhsl"
Oct 30 22:17:08.057025 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:08.056985    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c04bb48-1286-48fe-af0d-fc9ec478e663-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c04bb48-1286-48fe-af0d-fc9ec478e663-metrics-certs\") pod \"2c04bb48-1286-48fe-af0d-fc9ec478e663\" (UID: \"2c04bb48-1286-48fe-af0d-fc9ec478e663\") "
Oct 30 22:17:08.057237 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:08.057056    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c04bb48-1286-48fe-af0d-fc9ec478e663-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c04bb48-1286-48fe-af0d-fc9ec478e663-openshift-service-ca\") pod \"2c04bb48-1286-48fe-af0d-fc9ec478e663\" (UID: \"2c04bb48-1286-48fe-af0d-fc9ec478e663\") "
Oct 30 22:17:08.057237 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:08.057101    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c04bb48-1286-48fe-af0d-fc9ec478e663-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c04bb48-1286-48fe-af0d-fc9ec478e663-maas-api-tls\") pod \"2c04bb48-1286-48fe-af0d-fc9ec478e663\" (UID: \"2c04bb48-1286-48fe-af0d-fc9ec478e663\") "
Oct 30 22:17:08.057365 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:08.057263    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c04bb48-1286-48fe-af0d-fc9ec478e663-kube-api-access-6t4fs\" (UniqueName: \"kubernetes.io/projected/2c04bb48-1286-48fe-af0d-fc9ec478e663-kube-api-access-6t4fs\") pod \"2c04bb48-1286-48fe-af0d-fc9ec478e663\" (UID: \"2c04bb48-1286-48fe-af0d-fc9ec478e663\") "
Oct 30 22:17:08.058686 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:08.058655    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c04bb48-1286-48fe-af0d-fc9ec478e663-openshift-service-ca" pod "2c04bb48-1286-48fe-af0d-fc9ec478e663" (UID: "2c04bb48-1286-48fe-af0d-fc9ec478e663"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:17:08.060878 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:08.060852    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c04bb48-1286-48fe-af0d-fc9ec478e663-kube-api-access-6t4fs" pod "2c04bb48-1286-48fe-af0d-fc9ec478e663" (UID: "2c04bb48-1286-48fe-af0d-fc9ec478e663"). InnerVolumeSpecName "kube-api-access-6t4fs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:17:08.064016 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:08.063988    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c04bb48-1286-48fe-af0d-fc9ec478e663-maas-api-tls" pod "2c04bb48-1286-48fe-af0d-fc9ec478e663" (UID: "2c04bb48-1286-48fe-af0d-fc9ec478e663"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:17:08.064372 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:08.064339    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c04bb48-1286-48fe-af0d-fc9ec478e663-metrics-certs" pod "2c04bb48-1286-48fe-af0d-fc9ec478e663" (UID: "2c04bb48-1286-48fe-af0d-fc9ec478e663"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:17:08.159033 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:08.159002    5061 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c04bb48-1286-48fe-af0d-fc9ec478e663-metrics-certs\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:08.159033 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:08.159033    5061 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c04bb48-1286-48fe-af0d-fc9ec478e663-openshift-service-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:08.159242 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:08.159044    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c04bb48-1286-48fe-af0d-fc9ec478e663-maas-api-tls\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:08.159242 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:08.159053    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6t4fs\" (UniqueName: \"kubernetes.io/projected/2c04bb48-1286-48fe-af0d-fc9ec478e663-kube-api-access-6t4fs\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:08.411667 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:08.411626    5061 generic.go:356] "Generic (PLEG): container finished" podID="2c04bb48-1286-48fe-af0d-fc9ec478e663" containerID="4ec166df46a0650130217fff82aeceee87af729bcce56d41cc3dad535784fbd4" exitCode=0
Oct 30 22:17:08.412095 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:08.412068    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-411ff1-794dbcbd69-sfhsl" event={"ID":"2c04bb48-1286-48fe-af0d-fc9ec478e663","Type":"ContainerDied","Data":"4ec166df46a0650130217fff82aeceee87af729bcce56d41cc3dad535784fbd4"}
Oct 30 22:17:08.412327 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:08.412308    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-411ff1-794dbcbd69-sfhsl" event={"ID":"2c04bb48-1286-48fe-af0d-fc9ec478e663","Type":"ContainerDied","Data":"e5172fdc4012a4e44eff71d880a16081c647ae8c795a431f004b8ecf7f5b8e8b"}
Oct 30 22:17:08.412441 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:08.412428    5061 scope.go:122] "RemoveContainer" containerID="4ec166df46a0650130217fff82aeceee87af729bcce56d41cc3dad535784fbd4"
Oct 30 22:17:08.412719 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:08.412700    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-411ff1-794dbcbd69-sfhsl"
Oct 30 22:17:08.473054 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:08.472904    5061 scope.go:122] "RemoveContainer" containerID="4ec166df46a0650130217fff82aeceee87af729bcce56d41cc3dad535784fbd4"
Oct 30 22:17:08.476769 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:17:08.476718    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ec166df46a0650130217fff82aeceee87af729bcce56d41cc3dad535784fbd4\": container with ID starting with 4ec166df46a0650130217fff82aeceee87af729bcce56d41cc3dad535784fbd4 not found: ID does not exist" containerID="4ec166df46a0650130217fff82aeceee87af729bcce56d41cc3dad535784fbd4"
Oct 30 22:17:08.476912 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:08.476768    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ec166df46a0650130217fff82aeceee87af729bcce56d41cc3dad535784fbd4"} err="failed to get container status \"4ec166df46a0650130217fff82aeceee87af729bcce56d41cc3dad535784fbd4\": rpc error: code = NotFound desc = could not find container \"4ec166df46a0650130217fff82aeceee87af729bcce56d41cc3dad535784fbd4\": container with ID starting with 4ec166df46a0650130217fff82aeceee87af729bcce56d41cc3dad535784fbd4 not found: ID does not exist"
Oct 30 22:17:08.614014 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:08.610583    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-411ff1-794dbcbd69-sfhsl"]
Oct 30 22:17:08.623082 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:08.622994    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-411ff1-794dbcbd69-sfhsl"]
Oct 30 22:17:10.529186 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:10.529157    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c04bb48-1286-48fe-af0d-fc9ec478e663" path="/var/lib/kubelet/pods/2c04bb48-1286-48fe-af0d-fc9ec478e663/volumes"
Oct 30 22:17:14.286438 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:14.286405    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-263513-7c6995fd7b-4tgq8"]
Oct 30 22:17:14.387163 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:14.387133    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-263513-ccd8b86f5-5sphw"]
Oct 30 22:17:19.287197 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:19.287154    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-263513-7c6995fd7b-4tgq8" podUID="3fe1bb49-4bcc-49bd-9caa-70f83cbe8715" containerName="payload-processing" containerID="cri-o://444c6b807a34a63ca6c551ff9e73e90330ac21c9504e1320a400ce401cb81f46" gracePeriod=25
Oct 30 22:17:19.387434 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:19.387381    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-263513-ccd8b86f5-5sphw" podUID="ffa04d4a-c607-46a7-b4e0-c675ff341103" containerName="payload-pre-processing" containerID="cri-o://dcfd70860de19211172b564af88a4b0c6b19416ac141bd0b5d035f6b7c94a038" gracePeriod=25
Oct 30 22:17:19.500585 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:19.500498    5061 generic.go:356] "Generic (PLEG): container finished" podID="3fe1bb49-4bcc-49bd-9caa-70f83cbe8715" containerID="444c6b807a34a63ca6c551ff9e73e90330ac21c9504e1320a400ce401cb81f46" exitCode=0
Oct 30 22:17:19.500772 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:19.500720    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-263513-7c6995fd7b-4tgq8" event={"ID":"3fe1bb49-4bcc-49bd-9caa-70f83cbe8715","Type":"ContainerDied","Data":"444c6b807a34a63ca6c551ff9e73e90330ac21c9504e1320a400ce401cb81f46"}
Oct 30 22:17:19.579294 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:19.579179    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-263513-7c6995fd7b-4tgq8"
Oct 30 22:17:19.634322 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:19.634280    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fe1bb49-4bcc-49bd-9caa-70f83cbe8715-kube-api-access-hcbrq\" (UniqueName: \"kubernetes.io/projected/3fe1bb49-4bcc-49bd-9caa-70f83cbe8715-kube-api-access-hcbrq\") pod \"3fe1bb49-4bcc-49bd-9caa-70f83cbe8715\" (UID: \"3fe1bb49-4bcc-49bd-9caa-70f83cbe8715\") "
Oct 30 22:17:19.634517 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:19.634428    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3fe1bb49-4bcc-49bd-9caa-70f83cbe8715-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fe1bb49-4bcc-49bd-9caa-70f83cbe8715-plugins-config-volume\") pod \"3fe1bb49-4bcc-49bd-9caa-70f83cbe8715\" (UID: \"3fe1bb49-4bcc-49bd-9caa-70f83cbe8715\") "
Oct 30 22:17:19.635078 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:19.635054    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fe1bb49-4bcc-49bd-9caa-70f83cbe8715-plugins-config-volume" pod "3fe1bb49-4bcc-49bd-9caa-70f83cbe8715" (UID: "3fe1bb49-4bcc-49bd-9caa-70f83cbe8715"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:17:19.637679 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:19.637649    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fe1bb49-4bcc-49bd-9caa-70f83cbe8715-kube-api-access-hcbrq" pod "3fe1bb49-4bcc-49bd-9caa-70f83cbe8715" (UID: "3fe1bb49-4bcc-49bd-9caa-70f83cbe8715"). InnerVolumeSpecName "kube-api-access-hcbrq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:17:19.667216 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:19.667193    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-263513-ccd8b86f5-5sphw"
Oct 30 22:17:19.736227 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:19.736187    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffa04d4a-c607-46a7-b4e0-c675ff341103-kube-api-access-2b7t8\" (UniqueName: \"kubernetes.io/projected/ffa04d4a-c607-46a7-b4e0-c675ff341103-kube-api-access-2b7t8\") pod \"ffa04d4a-c607-46a7-b4e0-c675ff341103\" (UID: \"ffa04d4a-c607-46a7-b4e0-c675ff341103\") "
Oct 30 22:17:19.736465 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:19.736451    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffa04d4a-c607-46a7-b4e0-c675ff341103-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffa04d4a-c607-46a7-b4e0-c675ff341103-plugins-config-volume\") pod \"ffa04d4a-c607-46a7-b4e0-c675ff341103\" (UID: \"ffa04d4a-c607-46a7-b4e0-c675ff341103\") "
Oct 30 22:17:19.736873 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:19.736847    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffa04d4a-c607-46a7-b4e0-c675ff341103-plugins-config-volume" pod "ffa04d4a-c607-46a7-b4e0-c675ff341103" (UID: "ffa04d4a-c607-46a7-b4e0-c675ff341103"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:17:19.737038 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:19.737010    5061 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fe1bb49-4bcc-49bd-9caa-70f83cbe8715-plugins-config-volume\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:19.737108 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:19.737040    5061 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffa04d4a-c607-46a7-b4e0-c675ff341103-plugins-config-volume\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:19.737108 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:19.737051    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcbrq\" (UniqueName: \"kubernetes.io/projected/3fe1bb49-4bcc-49bd-9caa-70f83cbe8715-kube-api-access-hcbrq\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:19.738432 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:19.738404    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffa04d4a-c607-46a7-b4e0-c675ff341103-kube-api-access-2b7t8" pod "ffa04d4a-c607-46a7-b4e0-c675ff341103" (UID: "ffa04d4a-c607-46a7-b4e0-c675ff341103"). InnerVolumeSpecName "kube-api-access-2b7t8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:17:19.838343 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:19.838305    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2b7t8\" (UniqueName: \"kubernetes.io/projected/ffa04d4a-c607-46a7-b4e0-c675ff341103-kube-api-access-2b7t8\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:20.508902 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:20.508866    5061 generic.go:356] "Generic (PLEG): container finished" podID="ffa04d4a-c607-46a7-b4e0-c675ff341103" containerID="dcfd70860de19211172b564af88a4b0c6b19416ac141bd0b5d035f6b7c94a038" exitCode=0
Oct 30 22:17:20.509316 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:20.508929    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-263513-ccd8b86f5-5sphw"
Oct 30 22:17:20.509316 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:20.508948    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-263513-ccd8b86f5-5sphw" event={"ID":"ffa04d4a-c607-46a7-b4e0-c675ff341103","Type":"ContainerDied","Data":"dcfd70860de19211172b564af88a4b0c6b19416ac141bd0b5d035f6b7c94a038"}
Oct 30 22:17:20.509316 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:20.509029    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-263513-ccd8b86f5-5sphw" event={"ID":"ffa04d4a-c607-46a7-b4e0-c675ff341103","Type":"ContainerDied","Data":"e6d4871aa99ceefccc4b28f42dd7c91f51d64659fabaf335779116fb57ffc428"}
Oct 30 22:17:20.509316 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:20.509059    5061 scope.go:122] "RemoveContainer" containerID="dcfd70860de19211172b564af88a4b0c6b19416ac141bd0b5d035f6b7c94a038"
Oct 30 22:17:20.510594 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:20.510567    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-263513-7c6995fd7b-4tgq8" event={"ID":"3fe1bb49-4bcc-49bd-9caa-70f83cbe8715","Type":"ContainerDied","Data":"40b6a4dd7aba5c4f03bde365ae88bbd917a26bbe29323df5fbd6bcccefdaed8c"}
Oct 30 22:17:20.510698 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:20.510598    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-263513-7c6995fd7b-4tgq8"
Oct 30 22:17:20.524816 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:20.524795    5061 scope.go:122] "RemoveContainer" containerID="dcfd70860de19211172b564af88a4b0c6b19416ac141bd0b5d035f6b7c94a038"
Oct 30 22:17:20.525315 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:17:20.525170    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcfd70860de19211172b564af88a4b0c6b19416ac141bd0b5d035f6b7c94a038\": container with ID starting with dcfd70860de19211172b564af88a4b0c6b19416ac141bd0b5d035f6b7c94a038 not found: ID does not exist" containerID="dcfd70860de19211172b564af88a4b0c6b19416ac141bd0b5d035f6b7c94a038"
Oct 30 22:17:20.525315 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:20.525203    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcfd70860de19211172b564af88a4b0c6b19416ac141bd0b5d035f6b7c94a038"} err="failed to get container status \"dcfd70860de19211172b564af88a4b0c6b19416ac141bd0b5d035f6b7c94a038\": rpc error: code = NotFound desc = could not find container \"dcfd70860de19211172b564af88a4b0c6b19416ac141bd0b5d035f6b7c94a038\": container with ID starting with dcfd70860de19211172b564af88a4b0c6b19416ac141bd0b5d035f6b7c94a038 not found: ID does not exist"
Oct 30 22:17:20.525315 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:20.525224    5061 scope.go:122] "RemoveContainer" containerID="444c6b807a34a63ca6c551ff9e73e90330ac21c9504e1320a400ce401cb81f46"
Oct 30 22:17:20.666392 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:20.666312    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-263513-ccd8b86f5-5sphw"]
Oct 30 22:17:20.712937 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:20.712907    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-263513-ccd8b86f5-5sphw"]
Oct 30 22:17:20.816436 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:20.816402    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-263513-7c6995fd7b-4tgq8"]
Oct 30 22:17:20.820210 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:20.820183    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-263513-7c6995fd7b-4tgq8"]
Oct 30 22:17:22.529768 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:22.529738    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fe1bb49-4bcc-49bd-9caa-70f83cbe8715" path="/var/lib/kubelet/pods/3fe1bb49-4bcc-49bd-9caa-70f83cbe8715/volumes"
Oct 30 22:17:22.530190 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:22.530175    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffa04d4a-c607-46a7-b4e0-c675ff341103" path="/var/lib/kubelet/pods/ffa04d4a-c607-46a7-b4e0-c675ff341103/volumes"
Oct 30 22:17:24.667132 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:17:24.667100    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 22:17:26.836421 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:26.836392    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-sclrt_5e74071e-0462-42b5-a05c-d90bb638478a/console-operator/2.log"
Oct 30 22:17:26.865410 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:26.865384    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-sclrt_5e74071e-0462-42b5-a05c-d90bb638478a/console-operator/2.log"
Oct 30 22:17:29.043933 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:29.043898    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-6ea8da-4phr7"]
Oct 30 22:17:29.090467 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:29.090434    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-6ea8da-4phr7"]
Oct 30 22:17:30.529217 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:30.529185    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44d666c8-a3a5-4560-a446-ca4c8e4ecbca" path="/var/lib/kubelet/pods/44d666c8-a3a5-4560-a446-ca4c8e4ecbca/volumes"
Oct 30 22:17:36.647003 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:36.646960    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-c9dc10-pcbnw"]
Oct 30 22:17:36.647989 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:36.647966    5061 cpu_manager.go:414] "containerMap: removing container" podUID="2c04bb48-1286-48fe-af0d-fc9ec478e663" containerName="maas-api"
Oct 30 22:17:36.648098 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:36.647991    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c04bb48-1286-48fe-af0d-fc9ec478e663" containerName="maas-api"
Oct 30 22:17:36.648098 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:36.648006    5061 cpu_manager.go:414] "containerMap: removing container" podUID="3fe1bb49-4bcc-49bd-9caa-70f83cbe8715" containerName="payload-processing"
Oct 30 22:17:36.648098 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:36.648015    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fe1bb49-4bcc-49bd-9caa-70f83cbe8715" containerName="payload-processing"
Oct 30 22:17:36.648098 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:36.648063    5061 cpu_manager.go:414] "containerMap: removing container" podUID="4dfd4fc5-bd34-4451-9544-e3357405b66e" containerName="revoke-keys"
Oct 30 22:17:36.648098 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:36.648071    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4dfd4fc5-bd34-4451-9544-e3357405b66e" containerName="revoke-keys"
Oct 30 22:17:36.648263 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:36.648112    5061 cpu_manager.go:414] "containerMap: removing container" podUID="ffa04d4a-c607-46a7-b4e0-c675ff341103" containerName="payload-pre-processing"
Oct 30 22:17:36.648263 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:36.648119    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffa04d4a-c607-46a7-b4e0-c675ff341103" containerName="payload-pre-processing"
Oct 30 22:17:36.648329 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:36.648268    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c04bb48-1286-48fe-af0d-fc9ec478e663" containerName="maas-api"
Oct 30 22:17:36.648329 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:36.648281    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fe1bb49-4bcc-49bd-9caa-70f83cbe8715" containerName="payload-processing"
Oct 30 22:17:36.648329 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:36.648296    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="4dfd4fc5-bd34-4451-9544-e3357405b66e" containerName="revoke-keys"
Oct 30 22:17:36.648329 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:36.648310    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffa04d4a-c607-46a7-b4e0-c675ff341103" containerName="payload-pre-processing"
Oct 30 22:17:36.653860 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:36.653841    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-c9dc10-pcbnw"
Oct 30 22:17:36.684935 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:36.684907    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd7c97c5-1d79-47ea-a22b-eb6b5fd55ab6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-c9dc10-pcbnw\" (UID: \"bd7c97c5-1d79-47ea-a22b-eb6b5fd55ab6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-c9dc10-pcbnw"
Oct 30 22:17:36.685094 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:36.685076    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7r4fh\" (UniqueName: \"kubernetes.io/projected/bd7c97c5-1d79-47ea-a22b-eb6b5fd55ab6-kube-api-access-7r4fh\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-c9dc10-pcbnw\" (UID: \"bd7c97c5-1d79-47ea-a22b-eb6b5fd55ab6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-c9dc10-pcbnw"
Oct 30 22:17:36.685196 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:36.685180    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd7c97c5-1d79-47ea-a22b-eb6b5fd55ab6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-c9dc10-pcbnw\" (UID: \"bd7c97c5-1d79-47ea-a22b-eb6b5fd55ab6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-c9dc10-pcbnw"
Oct 30 22:17:36.706383 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:36.706356    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-c9dc10-pcbnw"]
Oct 30 22:17:36.749740 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:36.749710    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:17:36.749740 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:36.749734    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gdmfg\""
Oct 30 22:17:36.786771 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:36.786743    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7r4fh\" (UniqueName: \"kubernetes.io/projected/bd7c97c5-1d79-47ea-a22b-eb6b5fd55ab6-kube-api-access-7r4fh\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-c9dc10-pcbnw\" (UID: \"bd7c97c5-1d79-47ea-a22b-eb6b5fd55ab6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-c9dc10-pcbnw"
Oct 30 22:17:36.786899 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:36.786828    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd7c97c5-1d79-47ea-a22b-eb6b5fd55ab6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-c9dc10-pcbnw\" (UID: \"bd7c97c5-1d79-47ea-a22b-eb6b5fd55ab6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-c9dc10-pcbnw"
Oct 30 22:17:36.786899 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:36.786875    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd7c97c5-1d79-47ea-a22b-eb6b5fd55ab6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-c9dc10-pcbnw\" (UID: \"bd7c97c5-1d79-47ea-a22b-eb6b5fd55ab6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-c9dc10-pcbnw"
Oct 30 22:17:36.787711 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:36.787691    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd7c97c5-1d79-47ea-a22b-eb6b5fd55ab6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-c9dc10-pcbnw\" (UID: \"bd7c97c5-1d79-47ea-a22b-eb6b5fd55ab6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-c9dc10-pcbnw"
Oct 30 22:17:36.789157 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:36.789139    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd7c97c5-1d79-47ea-a22b-eb6b5fd55ab6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-c9dc10-pcbnw\" (UID: \"bd7c97c5-1d79-47ea-a22b-eb6b5fd55ab6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-c9dc10-pcbnw"
Oct 30 22:17:36.840573 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:36.840514    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7r4fh\" (UniqueName: \"kubernetes.io/projected/bd7c97c5-1d79-47ea-a22b-eb6b5fd55ab6-kube-api-access-7r4fh\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-c9dc10-pcbnw\" (UID: \"bd7c97c5-1d79-47ea-a22b-eb6b5fd55ab6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-c9dc10-pcbnw"
Oct 30 22:17:36.967570 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:36.967448    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-c9dc10-pcbnw"
Oct 30 22:17:37.347053 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:37.347019    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-c9dc10-pcbnw"]
Oct 30 22:17:37.348020 ip-10-0-130-60 kubenswrapper[5061]: W1030 22:17:37.347987    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd7c97c5_1d79_47ea_a22b_eb6b5fd55ab6.slice/crio-1b8b697c67c5a41b2374a5d3bbc0c3b03899305bd6e4cf032e8712dbc4e88398 WatchSource:0}: Error finding container 1b8b697c67c5a41b2374a5d3bbc0c3b03899305bd6e4cf032e8712dbc4e88398: Status 404 returned error can't find the container with id 1b8b697c67c5a41b2374a5d3bbc0c3b03899305bd6e4cf032e8712dbc4e88398
Oct 30 22:17:37.662383 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:37.662353    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-c9dc10-pcbnw" event={"ID":"bd7c97c5-1d79-47ea-a22b-eb6b5fd55ab6","Type":"ContainerStarted","Data":"754b30b1837ce0326dde1b1761f7480add843ae9295e3531650e5fd333f8152e"}
Oct 30 22:17:37.662808 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:37.662406    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-c9dc10-pcbnw" event={"ID":"bd7c97c5-1d79-47ea-a22b-eb6b5fd55ab6","Type":"ContainerStarted","Data":"1b8b697c67c5a41b2374a5d3bbc0c3b03899305bd6e4cf032e8712dbc4e88398"}
Oct 30 22:17:37.773642 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:37.773578    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-c9dc10-pcbnw" podStartSLOduration=1.773561146 podStartE2EDuration="1.773561146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:17:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:17:37.772469218 +0000 UTC m=+2111.385238976" watchObservedRunningTime="2026-10-30 22:17:37.773561146 +0000 UTC m=+2111.386330898"
Oct 30 22:17:38.670713 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:38.670619    5061 generic.go:356] "Generic (PLEG): container finished" podID="bd7c97c5-1d79-47ea-a22b-eb6b5fd55ab6" containerID="754b30b1837ce0326dde1b1761f7480add843ae9295e3531650e5fd333f8152e" exitCode=0
Oct 30 22:17:38.671089 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:38.670706    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-c9dc10-pcbnw" event={"ID":"bd7c97c5-1d79-47ea-a22b-eb6b5fd55ab6","Type":"ContainerDied","Data":"754b30b1837ce0326dde1b1761f7480add843ae9295e3531650e5fd333f8152e"}
Oct 30 22:17:39.837586 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:39.837550    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-c9dc10-pcbnw"
Oct 30 22:17:39.928601 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:39.928499    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd7c97c5-1d79-47ea-a22b-eb6b5fd55ab6-kube-api-access-7r4fh\" (UniqueName: \"kubernetes.io/projected/bd7c97c5-1d79-47ea-a22b-eb6b5fd55ab6-kube-api-access-7r4fh\") pod \"bd7c97c5-1d79-47ea-a22b-eb6b5fd55ab6\" (UID: \"bd7c97c5-1d79-47ea-a22b-eb6b5fd55ab6\") "
Oct 30 22:17:39.928601 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:39.928595    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd7c97c5-1d79-47ea-a22b-eb6b5fd55ab6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd7c97c5-1d79-47ea-a22b-eb6b5fd55ab6-maas-api-xks-ca\") pod \"bd7c97c5-1d79-47ea-a22b-eb6b5fd55ab6\" (UID: \"bd7c97c5-1d79-47ea-a22b-eb6b5fd55ab6\") "
Oct 30 22:17:39.928828 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:39.928632    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd7c97c5-1d79-47ea-a22b-eb6b5fd55ab6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd7c97c5-1d79-47ea-a22b-eb6b5fd55ab6-maas-api-openshift-ca\") pod \"bd7c97c5-1d79-47ea-a22b-eb6b5fd55ab6\" (UID: \"bd7c97c5-1d79-47ea-a22b-eb6b5fd55ab6\") "
Oct 30 22:17:39.929071 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:39.929040    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd7c97c5-1d79-47ea-a22b-eb6b5fd55ab6-maas-api-openshift-ca" pod "bd7c97c5-1d79-47ea-a22b-eb6b5fd55ab6" (UID: "bd7c97c5-1d79-47ea-a22b-eb6b5fd55ab6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:17:39.929237 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:39.929222    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd7c97c5-1d79-47ea-a22b-eb6b5fd55ab6-maas-api-openshift-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:39.930649 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:39.930626    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd7c97c5-1d79-47ea-a22b-eb6b5fd55ab6-maas-api-xks-ca" pod "bd7c97c5-1d79-47ea-a22b-eb6b5fd55ab6" (UID: "bd7c97c5-1d79-47ea-a22b-eb6b5fd55ab6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:17:39.930794 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:39.930778    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd7c97c5-1d79-47ea-a22b-eb6b5fd55ab6-kube-api-access-7r4fh" pod "bd7c97c5-1d79-47ea-a22b-eb6b5fd55ab6" (UID: "bd7c97c5-1d79-47ea-a22b-eb6b5fd55ab6"). InnerVolumeSpecName "kube-api-access-7r4fh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:17:40.030258 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:40.030222    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7r4fh\" (UniqueName: \"kubernetes.io/projected/bd7c97c5-1d79-47ea-a22b-eb6b5fd55ab6-kube-api-access-7r4fh\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:40.030258 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:40.030256    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd7c97c5-1d79-47ea-a22b-eb6b5fd55ab6-maas-api-xks-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:40.686874 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:40.686841    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-c9dc10-pcbnw"
Oct 30 22:17:40.687075 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:40.686880    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-c9dc10-pcbnw" event={"ID":"bd7c97c5-1d79-47ea-a22b-eb6b5fd55ab6","Type":"ContainerDied","Data":"1b8b697c67c5a41b2374a5d3bbc0c3b03899305bd6e4cf032e8712dbc4e88398"}
Oct 30 22:17:40.687075 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:40.686929    5061 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1b8b697c67c5a41b2374a5d3bbc0c3b03899305bd6e4cf032e8712dbc4e88398"
Oct 30 22:17:44.235927 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:44.235892    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-25de17-2kqvc"]
Oct 30 22:17:44.282620 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:44.282591    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-25de17-2kqvc"]
Oct 30 22:17:44.529771 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:44.529697    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8134f480-834a-41ff-ac9a-383a760aecf4" path="/var/lib/kubelet/pods/8134f480-834a-41ff-ac9a-383a760aecf4/volumes"
Oct 30 22:17:48.162361 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:48.162317    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-c9dc10-c94b4fcc5-5jrrn"]
Oct 30 22:17:51.021263 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:51.021225    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-25de17-openshift-default-8687bbf6cc-p9mf7"]
Oct 30 22:17:51.021748 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:51.021546    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-25de17-openshift-default-8687bbf6cc-p9mf7" podUID="f464e173-8ada-4978-967f-7897d5bc5823" containerName="istio-proxy" containerID="cri-o://33a76dbb7759715346ee34e1de9d391b3bfc30b2dc6d18aa051fbbfe65bc7177" gracePeriod=30
Oct 30 22:17:52.411467 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:52.411423    5061 patch_prober.go:29] interesting pod/e2e-api-a-25de17-openshift-default-8687bbf6cc-p9mf7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:17:52.411864 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:52.411507    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-25de17-openshift-default-8687bbf6cc-p9mf7" podUID="f464e173-8ada-4978-967f-7897d5bc5823" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:17:53.163636 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:53.163593    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-c9dc10-c94b4fcc5-5jrrn" podUID="64ca325f-739e-4a19-b117-8ddff188ea10" containerName="payload-pre-processing" containerID="cri-o://17260df1162d1e778a70113a6d4df9a6b12ba67df360c4b01832250d0c828e80" gracePeriod=25
Oct 30 22:17:53.437340 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:53.437310    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-c9dc10-c94b4fcc5-5jrrn"
Oct 30 22:17:53.550821 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:53.549429    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64ca325f-739e-4a19-b117-8ddff188ea10-kube-api-access-hq5zd\" (UniqueName: \"kubernetes.io/projected/64ca325f-739e-4a19-b117-8ddff188ea10-kube-api-access-hq5zd\") pod \"64ca325f-739e-4a19-b117-8ddff188ea10\" (UID: \"64ca325f-739e-4a19-b117-8ddff188ea10\") "
Oct 30 22:17:53.550821 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:53.549787    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64ca325f-739e-4a19-b117-8ddff188ea10-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64ca325f-739e-4a19-b117-8ddff188ea10-plugins-config-volume\") pod \"64ca325f-739e-4a19-b117-8ddff188ea10\" (UID: \"64ca325f-739e-4a19-b117-8ddff188ea10\") "
Oct 30 22:17:53.551666 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:53.551619    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64ca325f-739e-4a19-b117-8ddff188ea10-plugins-config-volume" pod "64ca325f-739e-4a19-b117-8ddff188ea10" (UID: "64ca325f-739e-4a19-b117-8ddff188ea10"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:17:53.563313 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:53.563257    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64ca325f-739e-4a19-b117-8ddff188ea10-kube-api-access-hq5zd" pod "64ca325f-739e-4a19-b117-8ddff188ea10" (UID: "64ca325f-739e-4a19-b117-8ddff188ea10"). InnerVolumeSpecName "kube-api-access-hq5zd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:17:53.652339 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:53.652236    5061 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64ca325f-739e-4a19-b117-8ddff188ea10-plugins-config-volume\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:53.652339 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:53.652279    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hq5zd\" (UniqueName: \"kubernetes.io/projected/64ca325f-739e-4a19-b117-8ddff188ea10-kube-api-access-hq5zd\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:53.899598 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:53.899558    5061 generic.go:356] "Generic (PLEG): container finished" podID="64ca325f-739e-4a19-b117-8ddff188ea10" containerID="17260df1162d1e778a70113a6d4df9a6b12ba67df360c4b01832250d0c828e80" exitCode=0
Oct 30 22:17:53.899819 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:53.899724    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-c9dc10-c94b4fcc5-5jrrn"
Oct 30 22:17:53.899819 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:53.899783    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-c9dc10-c94b4fcc5-5jrrn" event={"ID":"64ca325f-739e-4a19-b117-8ddff188ea10","Type":"ContainerDied","Data":"17260df1162d1e778a70113a6d4df9a6b12ba67df360c4b01832250d0c828e80"}
Oct 30 22:17:53.899961 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:53.899834    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-c9dc10-c94b4fcc5-5jrrn" event={"ID":"64ca325f-739e-4a19-b117-8ddff188ea10","Type":"ContainerDied","Data":"54ffffff7dc888205757060c44a67a323fbfa6f8df706c4ccad53d6f1aec887c"}
Oct 30 22:17:53.899961 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:53.899864    5061 scope.go:122] "RemoveContainer" containerID="17260df1162d1e778a70113a6d4df9a6b12ba67df360c4b01832250d0c828e80"
Oct 30 22:17:53.935264 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:53.935232    5061 scope.go:122] "RemoveContainer" containerID="17260df1162d1e778a70113a6d4df9a6b12ba67df360c4b01832250d0c828e80"
Oct 30 22:17:53.935886 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:17:53.935852    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17260df1162d1e778a70113a6d4df9a6b12ba67df360c4b01832250d0c828e80\": container with ID starting with 17260df1162d1e778a70113a6d4df9a6b12ba67df360c4b01832250d0c828e80 not found: ID does not exist" containerID="17260df1162d1e778a70113a6d4df9a6b12ba67df360c4b01832250d0c828e80"
Oct 30 22:17:53.936020 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:53.935901    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17260df1162d1e778a70113a6d4df9a6b12ba67df360c4b01832250d0c828e80"} err="failed to get container status \"17260df1162d1e778a70113a6d4df9a6b12ba67df360c4b01832250d0c828e80\": rpc error: code = NotFound desc = could not find container \"17260df1162d1e778a70113a6d4df9a6b12ba67df360c4b01832250d0c828e80\": container with ID starting with 17260df1162d1e778a70113a6d4df9a6b12ba67df360c4b01832250d0c828e80 not found: ID does not exist"
Oct 30 22:17:54.062174 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:54.062142    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-c9dc10-c94b4fcc5-5jrrn"]
Oct 30 22:17:54.065808 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:54.065778    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-c9dc10-c94b4fcc5-5jrrn"]
Oct 30 22:17:54.530647 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:54.530616    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64ca325f-739e-4a19-b117-8ddff188ea10" path="/var/lib/kubelet/pods/64ca325f-739e-4a19-b117-8ddff188ea10/volumes"
Oct 30 22:17:54.669219 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:17:54.669185    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 22:17:56.480287 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:56.480258    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-25de17-openshift-default-8687bbf6cc-p9mf7"
Oct 30 22:17:56.534598 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:56.534564    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f464e173-8ada-4978-967f-7897d5bc5823-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f464e173-8ada-4978-967f-7897d5bc5823-workload-certs\") pod \"f464e173-8ada-4978-967f-7897d5bc5823\" (UID: \"f464e173-8ada-4978-967f-7897d5bc5823\") "
Oct 30 22:17:56.534907 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:56.534879    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f464e173-8ada-4978-967f-7897d5bc5823-workload-certs" pod "f464e173-8ada-4978-967f-7897d5bc5823" (UID: "f464e173-8ada-4978-967f-7897d5bc5823"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:17:56.535032 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:56.534999    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f464e173-8ada-4978-967f-7897d5bc5823-credential-socket" pod "f464e173-8ada-4978-967f-7897d5bc5823" (UID: "f464e173-8ada-4978-967f-7897d5bc5823"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:17:56.535182 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:56.534714    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f464e173-8ada-4978-967f-7897d5bc5823-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f464e173-8ada-4978-967f-7897d5bc5823-credential-socket\") pod \"f464e173-8ada-4978-967f-7897d5bc5823\" (UID: \"f464e173-8ada-4978-967f-7897d5bc5823\") "
Oct 30 22:17:56.535284 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:56.535204    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f464e173-8ada-4978-967f-7897d5bc5823-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f464e173-8ada-4978-967f-7897d5bc5823-istio-podinfo\") pod \"f464e173-8ada-4978-967f-7897d5bc5823\" (UID: \"f464e173-8ada-4978-967f-7897d5bc5823\") "
Oct 30 22:17:56.535284 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:56.535236    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f464e173-8ada-4978-967f-7897d5bc5823-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f464e173-8ada-4978-967f-7897d5bc5823-istio-envoy\") pod \"f464e173-8ada-4978-967f-7897d5bc5823\" (UID: \"f464e173-8ada-4978-967f-7897d5bc5823\") "
Oct 30 22:17:56.535452 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:56.535303    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f464e173-8ada-4978-967f-7897d5bc5823-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f464e173-8ada-4978-967f-7897d5bc5823-istiod-ca-cert\") pod \"f464e173-8ada-4978-967f-7897d5bc5823\" (UID: \"f464e173-8ada-4978-967f-7897d5bc5823\") "
Oct 30 22:17:56.535452 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:56.535391    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f464e173-8ada-4978-967f-7897d5bc5823-istio-token\" (UniqueName: \"kubernetes.io/projected/f464e173-8ada-4978-967f-7897d5bc5823-istio-token\") pod \"f464e173-8ada-4978-967f-7897d5bc5823\" (UID: \"f464e173-8ada-4978-967f-7897d5bc5823\") "
Oct 30 22:17:56.535629 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:56.535475    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f464e173-8ada-4978-967f-7897d5bc5823-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f464e173-8ada-4978-967f-7897d5bc5823-istio-data\") pod \"f464e173-8ada-4978-967f-7897d5bc5823\" (UID: \"f464e173-8ada-4978-967f-7897d5bc5823\") "
Oct 30 22:17:56.535629 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:56.535555    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f464e173-8ada-4978-967f-7897d5bc5823-kube-api-access-cx2x4\" (UniqueName: \"kubernetes.io/projected/f464e173-8ada-4978-967f-7897d5bc5823-kube-api-access-cx2x4\") pod \"f464e173-8ada-4978-967f-7897d5bc5823\" (UID: \"f464e173-8ada-4978-967f-7897d5bc5823\") "
Oct 30 22:17:56.536213 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:56.536187    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f464e173-8ada-4978-967f-7897d5bc5823-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f464e173-8ada-4978-967f-7897d5bc5823-workload-socket\") pod \"f464e173-8ada-4978-967f-7897d5bc5823\" (UID: \"f464e173-8ada-4978-967f-7897d5bc5823\") "
Oct 30 22:17:56.536860 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:56.536700    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f464e173-8ada-4978-967f-7897d5bc5823-workload-socket" pod "f464e173-8ada-4978-967f-7897d5bc5823" (UID: "f464e173-8ada-4978-967f-7897d5bc5823"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:17:56.537065 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:56.536905    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f464e173-8ada-4978-967f-7897d5bc5823-istio-data" pod "f464e173-8ada-4978-967f-7897d5bc5823" (UID: "f464e173-8ada-4978-967f-7897d5bc5823"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:17:56.537647 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:56.537191    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f464e173-8ada-4978-967f-7897d5bc5823-istiod-ca-cert" pod "f464e173-8ada-4978-967f-7897d5bc5823" (UID: "f464e173-8ada-4978-967f-7897d5bc5823"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:17:56.538398 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:56.538377    5061 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f464e173-8ada-4978-967f-7897d5bc5823-istio-data\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:56.538518 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:56.538505    5061 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f464e173-8ada-4978-967f-7897d5bc5823-workload-socket\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:56.538705 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:56.538691    5061 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f464e173-8ada-4978-967f-7897d5bc5823-workload-certs\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:56.538792 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:56.538781    5061 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f464e173-8ada-4978-967f-7897d5bc5823-credential-socket\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:56.538873 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:56.538865    5061 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f464e173-8ada-4978-967f-7897d5bc5823-istiod-ca-cert\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:56.538939 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:56.538705    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f464e173-8ada-4978-967f-7897d5bc5823-istio-token" pod "f464e173-8ada-4978-967f-7897d5bc5823" (UID: "f464e173-8ada-4978-967f-7897d5bc5823"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:17:56.538998 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:56.538919    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f464e173-8ada-4978-967f-7897d5bc5823-istio-podinfo" pod "f464e173-8ada-4978-967f-7897d5bc5823" (UID: "f464e173-8ada-4978-967f-7897d5bc5823"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:17:56.541006 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:56.540982    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f464e173-8ada-4978-967f-7897d5bc5823-istio-envoy" pod "f464e173-8ada-4978-967f-7897d5bc5823" (UID: "f464e173-8ada-4978-967f-7897d5bc5823"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:17:56.541209 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:56.541182    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f464e173-8ada-4978-967f-7897d5bc5823-kube-api-access-cx2x4" pod "f464e173-8ada-4978-967f-7897d5bc5823" (UID: "f464e173-8ada-4978-967f-7897d5bc5823"). InnerVolumeSpecName "kube-api-access-cx2x4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:17:56.640389 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:56.640349    5061 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f464e173-8ada-4978-967f-7897d5bc5823-istio-podinfo\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:56.640389 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:56.640389    5061 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f464e173-8ada-4978-967f-7897d5bc5823-istio-envoy\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:56.640655 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:56.640407    5061 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f464e173-8ada-4978-967f-7897d5bc5823-istio-token\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:56.640655 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:56.640422    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cx2x4\" (UniqueName: \"kubernetes.io/projected/f464e173-8ada-4978-967f-7897d5bc5823-kube-api-access-cx2x4\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:56.927876 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:56.927839    5061 generic.go:356] "Generic (PLEG): container finished" podID="f464e173-8ada-4978-967f-7897d5bc5823" containerID="33a76dbb7759715346ee34e1de9d391b3bfc30b2dc6d18aa051fbbfe65bc7177" exitCode=0
Oct 30 22:17:56.928069 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:56.927896    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-25de17-openshift-default-8687bbf6cc-p9mf7"
Oct 30 22:17:56.928069 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:56.927922    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-25de17-openshift-default-8687bbf6cc-p9mf7" event={"ID":"f464e173-8ada-4978-967f-7897d5bc5823","Type":"ContainerDied","Data":"33a76dbb7759715346ee34e1de9d391b3bfc30b2dc6d18aa051fbbfe65bc7177"}
Oct 30 22:17:56.928069 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:56.927983    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-25de17-openshift-default-8687bbf6cc-p9mf7" event={"ID":"f464e173-8ada-4978-967f-7897d5bc5823","Type":"ContainerDied","Data":"a5a72e0acfa770b686d4c6a7d9583167b409e494c7c403e073be9f7f25ce58ca"}
Oct 30 22:17:56.928069 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:56.928007    5061 scope.go:122] "RemoveContainer" containerID="33a76dbb7759715346ee34e1de9d391b3bfc30b2dc6d18aa051fbbfe65bc7177"
Oct 30 22:17:56.943122 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:56.943099    5061 scope.go:122] "RemoveContainer" containerID="afb0f3ee46792d05f5b3ccc7c84b1d1ecdbe202f6943a3d166ff2be91f705a77"
Oct 30 22:17:56.958920 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:56.958894    5061 scope.go:122] "RemoveContainer" containerID="33a76dbb7759715346ee34e1de9d391b3bfc30b2dc6d18aa051fbbfe65bc7177"
Oct 30 22:17:56.960299 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:17:56.960177    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33a76dbb7759715346ee34e1de9d391b3bfc30b2dc6d18aa051fbbfe65bc7177\": container with ID starting with 33a76dbb7759715346ee34e1de9d391b3bfc30b2dc6d18aa051fbbfe65bc7177 not found: ID does not exist" containerID="33a76dbb7759715346ee34e1de9d391b3bfc30b2dc6d18aa051fbbfe65bc7177"
Oct 30 22:17:56.960299 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:56.960220    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33a76dbb7759715346ee34e1de9d391b3bfc30b2dc6d18aa051fbbfe65bc7177"} err="failed to get container status \"33a76dbb7759715346ee34e1de9d391b3bfc30b2dc6d18aa051fbbfe65bc7177\": rpc error: code = NotFound desc = could not find container \"33a76dbb7759715346ee34e1de9d391b3bfc30b2dc6d18aa051fbbfe65bc7177\": container with ID starting with 33a76dbb7759715346ee34e1de9d391b3bfc30b2dc6d18aa051fbbfe65bc7177 not found: ID does not exist"
Oct 30 22:17:56.960299 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:56.960238    5061 scope.go:122] "RemoveContainer" containerID="afb0f3ee46792d05f5b3ccc7c84b1d1ecdbe202f6943a3d166ff2be91f705a77"
Oct 30 22:17:56.960760 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:17:56.960549    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afb0f3ee46792d05f5b3ccc7c84b1d1ecdbe202f6943a3d166ff2be91f705a77\": container with ID starting with afb0f3ee46792d05f5b3ccc7c84b1d1ecdbe202f6943a3d166ff2be91f705a77 not found: ID does not exist" containerID="afb0f3ee46792d05f5b3ccc7c84b1d1ecdbe202f6943a3d166ff2be91f705a77"
Oct 30 22:17:56.960760 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:56.960579    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afb0f3ee46792d05f5b3ccc7c84b1d1ecdbe202f6943a3d166ff2be91f705a77"} err="failed to get container status \"afb0f3ee46792d05f5b3ccc7c84b1d1ecdbe202f6943a3d166ff2be91f705a77\": rpc error: code = NotFound desc = could not find container \"afb0f3ee46792d05f5b3ccc7c84b1d1ecdbe202f6943a3d166ff2be91f705a77\": container with ID starting with afb0f3ee46792d05f5b3ccc7c84b1d1ecdbe202f6943a3d166ff2be91f705a77 not found: ID does not exist"
Oct 30 22:17:57.088483 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:57.088446    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-25de17-openshift-default-8687bbf6cc-p9mf7"]
Oct 30 22:17:57.141979 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:57.141941    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-25de17-openshift-default-8687bbf6cc-p9mf7"]
Oct 30 22:17:58.531810 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:17:58.531773    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f464e173-8ada-4978-967f-7897d5bc5823" path="/var/lib/kubelet/pods/f464e173-8ada-4978-967f-7897d5bc5823/volumes"
Oct 30 22:18:04.572785 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:04.572750    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-778d39d3-d4gvh"]
Oct 30 22:18:04.617833 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:04.617750    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-778d39d3-d4gvh"]
Oct 30 22:18:06.531112 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:06.531084    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e7528ce-43ab-4e4c-be3f-b5a87b82d3b5" path="/var/lib/kubelet/pods/2e7528ce-43ab-4e4c-be3f-b5a87b82d3b5/volumes"
Oct 30 22:18:08.112994 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:08.112911    5061 generic.go:356] "Generic (PLEG): container finished" podID="0a5f5114-bc2f-48c9-af68-f2d981ec9ce5" containerID="c26f9976807884d9967ba246acee4e31c5f8093e873cdd5407d6f8f3cde6f64d" exitCode=0
Oct 30 22:18:08.112994 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:08.112982    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0ce563-openshift-default-f75cc7fc9-445r7" event={"ID":"0a5f5114-bc2f-48c9-af68-f2d981ec9ce5","Type":"ContainerDied","Data":"c26f9976807884d9967ba246acee4e31c5f8093e873cdd5407d6f8f3cde6f64d"}
Oct 30 22:18:08.113449 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:08.113383    5061 scope.go:122] "RemoveContainer" containerID="c26f9976807884d9967ba246acee4e31c5f8093e873cdd5407d6f8f3cde6f64d"
Oct 30 22:18:08.115028 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:08.114994    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:18:08.115149 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:08.115087    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:18:08.115223 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:08.115154    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:18:09.141887 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:09.141842    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0ce563-openshift-default-f75cc7fc9-445r7" event={"ID":"0a5f5114-bc2f-48c9-af68-f2d981ec9ce5","Type":"ContainerStarted","Data":"e44ff32b18531be8417e7c327ccefc08dc7776ab4ab4bc3efff775c162152b97"}
Oct 30 22:18:09.583406 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:09.583376    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-0ce563-openshift-default-f75cc7fc9-445r7"
Oct 30 22:18:09.583650 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:09.583629    5061 patch_prober.go:29] interesting pod/e2e-api-b-0ce563-openshift-default-f75cc7fc9-445r7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body=
Oct 30 22:18:09.583704 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:09.583677    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0ce563-openshift-default-f75cc7fc9-445r7" podUID="0a5f5114-bc2f-48c9-af68-f2d981ec9ce5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused"
Oct 30 22:18:10.583653 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:10.583617    5061 patch_prober.go:29] interesting pod/e2e-api-b-0ce563-openshift-default-f75cc7fc9-445r7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body=
Oct 30 22:18:10.584062 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:10.583684    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0ce563-openshift-default-f75cc7fc9-445r7" podUID="0a5f5114-bc2f-48c9-af68-f2d981ec9ce5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused"
Oct 30 22:18:11.583865 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:11.583827    5061 patch_prober.go:29] interesting pod/e2e-api-b-0ce563-openshift-default-f75cc7fc9-445r7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body=
Oct 30 22:18:11.584271 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:11.583880    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0ce563-openshift-default-f75cc7fc9-445r7" podUID="0a5f5114-bc2f-48c9-af68-f2d981ec9ce5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused"
Oct 30 22:18:12.583751 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:12.583717    5061 patch_prober.go:29] interesting pod/e2e-api-b-0ce563-openshift-default-f75cc7fc9-445r7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body=
Oct 30 22:18:12.583933 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:12.583768    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0ce563-openshift-default-f75cc7fc9-445r7" podUID="0a5f5114-bc2f-48c9-af68-f2d981ec9ce5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused"
Oct 30 22:18:13.584459 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:13.584417    5061 patch_prober.go:29] interesting pod/e2e-api-b-0ce563-openshift-default-f75cc7fc9-445r7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body=
Oct 30 22:18:13.584886 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:13.584486    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0ce563-openshift-default-f75cc7fc9-445r7" podUID="0a5f5114-bc2f-48c9-af68-f2d981ec9ce5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused"
Oct 30 22:18:14.583855 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:14.583816    5061 patch_prober.go:29] interesting pod/e2e-api-b-0ce563-openshift-default-f75cc7fc9-445r7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body=
Oct 30 22:18:14.584045 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:14.583888    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0ce563-openshift-default-f75cc7fc9-445r7" podUID="0a5f5114-bc2f-48c9-af68-f2d981ec9ce5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused"
Oct 30 22:18:15.588247 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:15.588213    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-0ce563-openshift-default-f75cc7fc9-445r7"
Oct 30 22:18:15.588730 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:15.588640    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-0ce563-openshift-default-f75cc7fc9-445r7"
Oct 30 22:18:15.589300 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:15.589277    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-0ce563-openshift-default-f75cc7fc9-445r7"
Oct 30 22:18:23.088391 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:23.088354    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-263513-7c96866b48-56jwr"]
Oct 30 22:18:23.088876 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:23.088552    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-263513-7c96866b48-56jwr" podUID="49d78ffc-f174-411c-aae5-20b1c79f90d2" containerName="maas-api" containerID="cri-o://780fe6097849a8777aaed56649eec3c84f863676a9ea042c847062b87bc8bf0b" gracePeriod=30
Oct 30 22:18:23.251032 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:23.250994    5061 generic.go:356] "Generic (PLEG): container finished" podID="49d78ffc-f174-411c-aae5-20b1c79f90d2" containerID="780fe6097849a8777aaed56649eec3c84f863676a9ea042c847062b87bc8bf0b" exitCode=0
Oct 30 22:18:23.251219 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:23.251069    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-263513-7c96866b48-56jwr" event={"ID":"49d78ffc-f174-411c-aae5-20b1c79f90d2","Type":"ContainerDied","Data":"780fe6097849a8777aaed56649eec3c84f863676a9ea042c847062b87bc8bf0b"}
Oct 30 22:18:23.405100 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:23.405071    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-263513-7c96866b48-56jwr"
Oct 30 22:18:23.523018 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:23.520225    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/49d78ffc-f174-411c-aae5-20b1c79f90d2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49d78ffc-f174-411c-aae5-20b1c79f90d2-maas-api-tls\") pod \"49d78ffc-f174-411c-aae5-20b1c79f90d2\" (UID: \"49d78ffc-f174-411c-aae5-20b1c79f90d2\") "
Oct 30 22:18:23.523018 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:23.520306    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/49d78ffc-f174-411c-aae5-20b1c79f90d2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/49d78ffc-f174-411c-aae5-20b1c79f90d2-metrics-certs\") pod \"49d78ffc-f174-411c-aae5-20b1c79f90d2\" (UID: \"49d78ffc-f174-411c-aae5-20b1c79f90d2\") "
Oct 30 22:18:23.523018 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:23.520369    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49d78ffc-f174-411c-aae5-20b1c79f90d2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49d78ffc-f174-411c-aae5-20b1c79f90d2-openshift-service-ca\") pod \"49d78ffc-f174-411c-aae5-20b1c79f90d2\" (UID: \"49d78ffc-f174-411c-aae5-20b1c79f90d2\") "
Oct 30 22:18:23.523018 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:23.520403    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49d78ffc-f174-411c-aae5-20b1c79f90d2-kube-api-access-877f9\" (UniqueName: \"kubernetes.io/projected/49d78ffc-f174-411c-aae5-20b1c79f90d2-kube-api-access-877f9\") pod \"49d78ffc-f174-411c-aae5-20b1c79f90d2\" (UID: \"49d78ffc-f174-411c-aae5-20b1c79f90d2\") "
Oct 30 22:18:23.523018 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:23.521882    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49d78ffc-f174-411c-aae5-20b1c79f90d2-openshift-service-ca" pod "49d78ffc-f174-411c-aae5-20b1c79f90d2" (UID: "49d78ffc-f174-411c-aae5-20b1c79f90d2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:18:23.529048 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:23.529009    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49d78ffc-f174-411c-aae5-20b1c79f90d2-metrics-certs" pod "49d78ffc-f174-411c-aae5-20b1c79f90d2" (UID: "49d78ffc-f174-411c-aae5-20b1c79f90d2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:18:23.531947 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:23.531917    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49d78ffc-f174-411c-aae5-20b1c79f90d2-kube-api-access-877f9" pod "49d78ffc-f174-411c-aae5-20b1c79f90d2" (UID: "49d78ffc-f174-411c-aae5-20b1c79f90d2"). InnerVolumeSpecName "kube-api-access-877f9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:18:23.537676 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:23.537634    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49d78ffc-f174-411c-aae5-20b1c79f90d2-maas-api-tls" pod "49d78ffc-f174-411c-aae5-20b1c79f90d2" (UID: "49d78ffc-f174-411c-aae5-20b1c79f90d2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:18:23.622961 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:23.622866    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49d78ffc-f174-411c-aae5-20b1c79f90d2-maas-api-tls\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:23.622961 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:23.622902    5061 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/49d78ffc-f174-411c-aae5-20b1c79f90d2-metrics-certs\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:23.622961 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:23.622917    5061 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49d78ffc-f174-411c-aae5-20b1c79f90d2-openshift-service-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:23.622961 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:23.622933    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-877f9\" (UniqueName: \"kubernetes.io/projected/49d78ffc-f174-411c-aae5-20b1c79f90d2-kube-api-access-877f9\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:24.260826 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:24.260792    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-263513-7c96866b48-56jwr"
Oct 30 22:18:24.260826 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:24.260802    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-263513-7c96866b48-56jwr" event={"ID":"49d78ffc-f174-411c-aae5-20b1c79f90d2","Type":"ContainerDied","Data":"e07b88be84f7cbd00144118818f16a40c49794dc8c00487d0a2c107c20afd6f9"}
Oct 30 22:18:24.261359 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:24.260861    5061 scope.go:122] "RemoveContainer" containerID="780fe6097849a8777aaed56649eec3c84f863676a9ea042c847062b87bc8bf0b"
Oct 30 22:18:24.406639 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:18:24.406606    5061 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49d78ffc_f174_411c_aae5_20b1c79f90d2.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 22:18:24.448055 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:24.448013    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-263513-7c96866b48-56jwr"]
Oct 30 22:18:24.496036 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:24.495989    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-263513-7c96866b48-56jwr"]
Oct 30 22:18:24.556360 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:24.556324    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49d78ffc-f174-411c-aae5-20b1c79f90d2" path="/var/lib/kubelet/pods/49d78ffc-f174-411c-aae5-20b1c79f90d2/volumes"
Oct 30 22:18:24.733552 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:18:24.726112    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 22:18:24.938889 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:24.938794    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-3aa85e-7865f7fb9f-cf7p2"]
Oct 30 22:18:24.939715 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:24.939689    5061 cpu_manager.go:414] "containerMap: removing container" podUID="f464e173-8ada-4978-967f-7897d5bc5823" containerName="istio-proxy"
Oct 30 22:18:24.939715 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:24.939712    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f464e173-8ada-4978-967f-7897d5bc5823" containerName="istio-proxy"
Oct 30 22:18:24.939903 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:24.939728    5061 cpu_manager.go:414] "containerMap: removing container" podUID="49d78ffc-f174-411c-aae5-20b1c79f90d2" containerName="maas-api"
Oct 30 22:18:24.939903 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:24.939733    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49d78ffc-f174-411c-aae5-20b1c79f90d2" containerName="maas-api"
Oct 30 22:18:24.939903 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:24.939747    5061 cpu_manager.go:414] "containerMap: removing container" podUID="bd7c97c5-1d79-47ea-a22b-eb6b5fd55ab6" containerName="revoke-keys"
Oct 30 22:18:24.939903 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:24.939756    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd7c97c5-1d79-47ea-a22b-eb6b5fd55ab6" containerName="revoke-keys"
Oct 30 22:18:24.939903 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:24.939768    5061 cpu_manager.go:414] "containerMap: removing container" podUID="f464e173-8ada-4978-967f-7897d5bc5823" containerName="istio-proxy"
Oct 30 22:18:24.939903 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:24.939773    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f464e173-8ada-4978-967f-7897d5bc5823" containerName="istio-proxy"
Oct 30 22:18:24.939903 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:24.939803    5061 cpu_manager.go:414] "containerMap: removing container" podUID="64ca325f-739e-4a19-b117-8ddff188ea10" containerName="payload-pre-processing"
Oct 30 22:18:24.939903 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:24.939808    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64ca325f-739e-4a19-b117-8ddff188ea10" containerName="payload-pre-processing"
Oct 30 22:18:24.940164 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:24.939930    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="f464e173-8ada-4978-967f-7897d5bc5823" containerName="istio-proxy"
Oct 30 22:18:24.940164 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:24.939939    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="49d78ffc-f174-411c-aae5-20b1c79f90d2" containerName="maas-api"
Oct 30 22:18:24.940164 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:24.939945    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd7c97c5-1d79-47ea-a22b-eb6b5fd55ab6" containerName="revoke-keys"
Oct 30 22:18:24.940164 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:24.939956    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="64ca325f-739e-4a19-b117-8ddff188ea10" containerName="payload-pre-processing"
Oct 30 22:18:24.945034 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:24.945007    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3aa85e-7865f7fb9f-cf7p2"
Oct 30 22:18:25.003076 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:25.003044    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-3aa85e\""
Oct 30 22:18:25.003591 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:25.003574    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-3aa85e-dockercfg-zgtkw\""
Oct 30 22:18:25.004887 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:25.004863    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-3aa85e-7865f7fb9f-cf7p2"]
Oct 30 22:18:25.050775 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:25.050740    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/046f4c6f-bc56-4b85-af3e-f30e75964175-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-3aa85e-7865f7fb9f-cf7p2\" (UID: \"046f4c6f-bc56-4b85-af3e-f30e75964175\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3aa85e-7865f7fb9f-cf7p2"
Oct 30 22:18:25.050984 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:25.050892    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cb5pt\" (UniqueName: \"kubernetes.io/projected/046f4c6f-bc56-4b85-af3e-f30e75964175-kube-api-access-cb5pt\") pod \"payload-pre-processing-e2e-inf-a-3aa85e-7865f7fb9f-cf7p2\" (UID: \"046f4c6f-bc56-4b85-af3e-f30e75964175\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3aa85e-7865f7fb9f-cf7p2"
Oct 30 22:18:25.152639 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:25.152601    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/046f4c6f-bc56-4b85-af3e-f30e75964175-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-3aa85e-7865f7fb9f-cf7p2\" (UID: \"046f4c6f-bc56-4b85-af3e-f30e75964175\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3aa85e-7865f7fb9f-cf7p2"
Oct 30 22:18:25.152853 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:25.152665    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cb5pt\" (UniqueName: \"kubernetes.io/projected/046f4c6f-bc56-4b85-af3e-f30e75964175-kube-api-access-cb5pt\") pod \"payload-pre-processing-e2e-inf-a-3aa85e-7865f7fb9f-cf7p2\" (UID: \"046f4c6f-bc56-4b85-af3e-f30e75964175\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3aa85e-7865f7fb9f-cf7p2"
Oct 30 22:18:25.153314 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:25.153294    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/046f4c6f-bc56-4b85-af3e-f30e75964175-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-3aa85e-7865f7fb9f-cf7p2\" (UID: \"046f4c6f-bc56-4b85-af3e-f30e75964175\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3aa85e-7865f7fb9f-cf7p2"
Oct 30 22:18:25.210424 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:25.210352    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cb5pt\" (UniqueName: \"kubernetes.io/projected/046f4c6f-bc56-4b85-af3e-f30e75964175-kube-api-access-cb5pt\") pod \"payload-pre-processing-e2e-inf-a-3aa85e-7865f7fb9f-cf7p2\" (UID: \"046f4c6f-bc56-4b85-af3e-f30e75964175\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3aa85e-7865f7fb9f-cf7p2"
Oct 30 22:18:25.261407 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:25.261080    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3aa85e-7865f7fb9f-cf7p2"
Oct 30 22:18:25.645555 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:25.645512    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-3aa85e-7865f7fb9f-cf7p2"]
Oct 30 22:18:25.645874 ip-10-0-130-60 kubenswrapper[5061]: W1030 22:18:25.645844    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod046f4c6f_bc56_4b85_af3e_f30e75964175.slice/crio-a51ee3229a475292cef484d040ae3c046ec4bbc57f2fe3e58cbbc3c97fc7d866 WatchSource:0}: Error finding container a51ee3229a475292cef484d040ae3c046ec4bbc57f2fe3e58cbbc3c97fc7d866: Status 404 returned error can't find the container with id a51ee3229a475292cef484d040ae3c046ec4bbc57f2fe3e58cbbc3c97fc7d866
Oct 30 22:18:25.809608 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:25.809569    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-263513-hc7m9"]
Oct 30 22:18:25.813422 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:25.813392    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-263513-hc7m9"]
Oct 30 22:18:26.282054 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:26.282021    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3aa85e-7865f7fb9f-cf7p2" event={"ID":"046f4c6f-bc56-4b85-af3e-f30e75964175","Type":"ContainerStarted","Data":"70ad6d5ec123f4f70fe6569ea774ff104f1e2f3edbae025eaabfe8d3855e3e02"}
Oct 30 22:18:26.282493 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:26.282081    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3aa85e-7865f7fb9f-cf7p2" event={"ID":"046f4c6f-bc56-4b85-af3e-f30e75964175","Type":"ContainerStarted","Data":"a51ee3229a475292cef484d040ae3c046ec4bbc57f2fe3e58cbbc3c97fc7d866"}
Oct 30 22:18:26.282493 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:26.282110    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3aa85e-7865f7fb9f-cf7p2"
Oct 30 22:18:26.394858 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:26.394804    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3aa85e-7865f7fb9f-cf7p2" podStartSLOduration=2.3947878559999998 podStartE2EDuration="2.394787856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:18:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:18:26.392141199 +0000 UTC m=+2160.004910957" watchObservedRunningTime="2026-10-30 22:18:26.394787856 +0000 UTC m=+2160.007557640"
Oct 30 22:18:26.545567 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:26.545518    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4dfd4fc5-bd34-4451-9544-e3357405b66e" path="/var/lib/kubelet/pods/4dfd4fc5-bd34-4451-9544-e3357405b66e/volumes"
Oct 30 22:18:27.247885 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:27.247849    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-778d39d3-544ffbcf87-xvhr4"]
Oct 30 22:18:27.248777 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:27.248756    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="f464e173-8ada-4978-967f-7897d5bc5823" containerName="istio-proxy"
Oct 30 22:18:27.252572 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:27.252555    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-778d39d3-544ffbcf87-xvhr4"
Oct 30 22:18:27.368549 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:27.365939    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-778d39d3\""
Oct 30 22:18:27.368549 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:27.366059    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-778d39d3-dockercfg-v9svl\""
Oct 30 22:18:27.423553 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:27.422585    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-778d39d3-544ffbcf87-xvhr4"]
Oct 30 22:18:27.430553 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:27.429060    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tnm4\" (UniqueName: \"kubernetes.io/projected/9a858f67-af17-429f-9b06-71d5a7c3a7bf-kube-api-access-6tnm4\") pod \"payload-pre-processing-e2e-mt-778d39d3-544ffbcf87-xvhr4\" (UID: \"9a858f67-af17-429f-9b06-71d5a7c3a7bf\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-778d39d3-544ffbcf87-xvhr4"
Oct 30 22:18:27.430553 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:27.429199    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a858f67-af17-429f-9b06-71d5a7c3a7bf-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-778d39d3-544ffbcf87-xvhr4\" (UID: \"9a858f67-af17-429f-9b06-71d5a7c3a7bf\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-778d39d3-544ffbcf87-xvhr4"
Oct 30 22:18:27.530897 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:27.530807    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tnm4\" (UniqueName: \"kubernetes.io/projected/9a858f67-af17-429f-9b06-71d5a7c3a7bf-kube-api-access-6tnm4\") pod \"payload-pre-processing-e2e-mt-778d39d3-544ffbcf87-xvhr4\" (UID: \"9a858f67-af17-429f-9b06-71d5a7c3a7bf\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-778d39d3-544ffbcf87-xvhr4"
Oct 30 22:18:27.531073 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:27.530954    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a858f67-af17-429f-9b06-71d5a7c3a7bf-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-778d39d3-544ffbcf87-xvhr4\" (UID: \"9a858f67-af17-429f-9b06-71d5a7c3a7bf\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-778d39d3-544ffbcf87-xvhr4"
Oct 30 22:18:27.531624 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:27.531601    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a858f67-af17-429f-9b06-71d5a7c3a7bf-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-778d39d3-544ffbcf87-xvhr4\" (UID: \"9a858f67-af17-429f-9b06-71d5a7c3a7bf\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-778d39d3-544ffbcf87-xvhr4"
Oct 30 22:18:27.585058 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:27.585032    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tnm4\" (UniqueName: \"kubernetes.io/projected/9a858f67-af17-429f-9b06-71d5a7c3a7bf-kube-api-access-6tnm4\") pod \"payload-pre-processing-e2e-mt-778d39d3-544ffbcf87-xvhr4\" (UID: \"9a858f67-af17-429f-9b06-71d5a7c3a7bf\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-778d39d3-544ffbcf87-xvhr4"
Oct 30 22:18:27.866245 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:27.866211    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-778d39d3-544ffbcf87-xvhr4"
Oct 30 22:18:28.237657 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:28.237627    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-778d39d3-544ffbcf87-xvhr4"]
Oct 30 22:18:28.238738 ip-10-0-130-60 kubenswrapper[5061]: W1030 22:18:28.238707    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a858f67_af17_429f_9b06_71d5a7c3a7bf.slice/crio-4f5c94c3ae6cb8d62ad7a869da53990ff0b55ce385fce3fb4dd83e95596f5047 WatchSource:0}: Error finding container 4f5c94c3ae6cb8d62ad7a869da53990ff0b55ce385fce3fb4dd83e95596f5047: Status 404 returned error can't find the container with id 4f5c94c3ae6cb8d62ad7a869da53990ff0b55ce385fce3fb4dd83e95596f5047
Oct 30 22:18:28.299439 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:28.299400    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-778d39d3-544ffbcf87-xvhr4" event={"ID":"9a858f67-af17-429f-9b06-71d5a7c3a7bf","Type":"ContainerStarted","Data":"4f5c94c3ae6cb8d62ad7a869da53990ff0b55ce385fce3fb4dd83e95596f5047"}
Oct 30 22:18:29.311547 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:29.309500    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-778d39d3-544ffbcf87-xvhr4" event={"ID":"9a858f67-af17-429f-9b06-71d5a7c3a7bf","Type":"ContainerStarted","Data":"6de424961288f6e231abfcebd00a2f0151285170297fd818e3ad9801a3d1c39e"}
Oct 30 22:18:29.311547 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:29.309699    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-778d39d3-544ffbcf87-xvhr4"
Oct 30 22:18:29.432944 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:29.432864    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-778d39d3-544ffbcf87-xvhr4" podStartSLOduration=2.432838641 podStartE2EDuration="2.432838641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:18:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:18:29.419999752 +0000 UTC m=+2163.032769603" watchObservedRunningTime="2026-10-30 22:18:29.432838641 +0000 UTC m=+2163.045608401"
Oct 30 22:18:31.378348 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:31.378306    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-411ff1-lgs7j"]
Oct 30 22:18:31.433712 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:31.433676    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-411ff1-lgs7j"]
Oct 30 22:18:32.540486 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:32.540449    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84c958fa-2860-4399-918e-cbd96d83ccfe" path="/var/lib/kubelet/pods/84c958fa-2860-4399-918e-cbd96d83ccfe/volumes"
Oct 30 22:18:37.293152 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:37.293122    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3aa85e-7865f7fb9f-cf7p2"
Oct 30 22:18:39.412945 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:39.412912    5061 generic.go:356] "Generic (PLEG): container finished" podID="2cf14933-e261-4155-8a97-1a1a707c3550" containerID="aabf3da76eb49cc69fbbc7a4c143e478fb9db6b1c977cd981438852e705206a5" exitCode=0
Oct 30 22:18:39.413389 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:39.412981    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c1f64a-openshift-default-7d9c46ff4f-pnrz7" event={"ID":"2cf14933-e261-4155-8a97-1a1a707c3550","Type":"ContainerDied","Data":"aabf3da76eb49cc69fbbc7a4c143e478fb9db6b1c977cd981438852e705206a5"}
Oct 30 22:18:39.413434 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:39.413408    5061 scope.go:122] "RemoveContainer" containerID="aabf3da76eb49cc69fbbc7a4c143e478fb9db6b1c977cd981438852e705206a5"
Oct 30 22:18:39.415352 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:39.415323    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:18:39.415476 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:39.415398    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:18:39.415476 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:39.415432    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:18:40.241549 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:40.241508    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-f776cb/distinct-2-w3-f776cb-kserve-7b4d4cc954-d88pf"]
Oct 30 22:18:40.344910 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:40.344882    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-778d39d3-544ffbcf87-xvhr4"
Oct 30 22:18:40.440774 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:40.440736    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c1f64a-openshift-default-7d9c46ff4f-pnrz7" event={"ID":"2cf14933-e261-4155-8a97-1a1a707c3550","Type":"ContainerStarted","Data":"1f5800271acd94f8cdebe0c248c72e85a28f1512066737f0054574e478103aef"}
Oct 30 22:18:41.383604 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:41.383569    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-c1f64a-openshift-default-7d9c46ff4f-pnrz7"
Oct 30 22:18:41.384717 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:41.384693    5061 patch_prober.go:29] interesting pod/e2e-inf-a-c1f64a-openshift-default-7d9c46ff4f-pnrz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body=
Oct 30 22:18:41.384832 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:41.384754    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c1f64a-openshift-default-7d9c46ff4f-pnrz7" podUID="2cf14933-e261-4155-8a97-1a1a707c3550" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused"
Oct 30 22:18:42.384018 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:42.383973    5061 patch_prober.go:29] interesting pod/e2e-inf-a-c1f64a-openshift-default-7d9c46ff4f-pnrz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body=
Oct 30 22:18:42.384414 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:42.384045    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c1f64a-openshift-default-7d9c46ff4f-pnrz7" podUID="2cf14933-e261-4155-8a97-1a1a707c3550" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused"
Oct 30 22:18:43.384782 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:43.384741    5061 patch_prober.go:29] interesting pod/e2e-inf-a-c1f64a-openshift-default-7d9c46ff4f-pnrz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body=
Oct 30 22:18:43.385413 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:43.385382    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c1f64a-openshift-default-7d9c46ff4f-pnrz7" podUID="2cf14933-e261-4155-8a97-1a1a707c3550" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused"
Oct 30 22:18:44.384103 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:44.384064    5061 patch_prober.go:29] interesting pod/e2e-inf-a-c1f64a-openshift-default-7d9c46ff4f-pnrz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body=
Oct 30 22:18:44.384343 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:44.384121    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c1f64a-openshift-default-7d9c46ff4f-pnrz7" podUID="2cf14933-e261-4155-8a97-1a1a707c3550" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused"
Oct 30 22:18:45.384155 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:45.384112    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-c1f64a-openshift-default-7d9c46ff4f-pnrz7"
Oct 30 22:18:45.384593 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:45.384398    5061 patch_prober.go:29] interesting pod/e2e-inf-a-c1f64a-openshift-default-7d9c46ff4f-pnrz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body=
Oct 30 22:18:45.384593 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:45.384455    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c1f64a-openshift-default-7d9c46ff4f-pnrz7" podUID="2cf14933-e261-4155-8a97-1a1a707c3550" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused"
Oct 30 22:18:45.438474 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:45.438433    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-zskrq"]
Oct 30 22:18:45.438817 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:45.438784    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-zskrq" podUID="5eeacb87-9f34-4314-80f6-d538c5e843fa" containerName="istio-proxy" containerID="cri-o://14a3d242c861fb7cbe67ebdf86ad5e13a006edd2a1df573e31953065dc78cbbd" gracePeriod=30
Oct 30 22:18:46.388402 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:46.388353    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-c1f64a-openshift-default-7d9c46ff4f-pnrz7"
Oct 30 22:18:46.389563 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:46.389514    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-c1f64a-openshift-default-7d9c46ff4f-pnrz7"
Oct 30 22:18:47.270146 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:47.270112    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-f776cb-74d8d8c899-hnzj8"]
Oct 30 22:18:50.718949 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:50.718886    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-zskrq"
Oct 30 22:18:50.833713 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:50.833680    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5eeacb87-9f34-4314-80f6-d538c5e843fa-istio-token\" (UniqueName: \"kubernetes.io/projected/5eeacb87-9f34-4314-80f6-d538c5e843fa-istio-token\") pod \"5eeacb87-9f34-4314-80f6-d538c5e843fa\" (UID: \"5eeacb87-9f34-4314-80f6-d538c5e843fa\") "
Oct 30 22:18:50.833907 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:50.833741    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5eeacb87-9f34-4314-80f6-d538c5e843fa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eeacb87-9f34-4314-80f6-d538c5e843fa-credential-socket\") pod \"5eeacb87-9f34-4314-80f6-d538c5e843fa\" (UID: \"5eeacb87-9f34-4314-80f6-d538c5e843fa\") "
Oct 30 22:18:50.833907 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:50.833842    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5eeacb87-9f34-4314-80f6-d538c5e843fa-kube-api-access-4z24f\" (UniqueName: \"kubernetes.io/projected/5eeacb87-9f34-4314-80f6-d538c5e843fa-kube-api-access-4z24f\") pod \"5eeacb87-9f34-4314-80f6-d538c5e843fa\" (UID: \"5eeacb87-9f34-4314-80f6-d538c5e843fa\") "
Oct 30 22:18:50.833907 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:50.833892    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5eeacb87-9f34-4314-80f6-d538c5e843fa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5eeacb87-9f34-4314-80f6-d538c5e843fa-istio-podinfo\") pod \"5eeacb87-9f34-4314-80f6-d538c5e843fa\" (UID: \"5eeacb87-9f34-4314-80f6-d538c5e843fa\") "
Oct 30 22:18:50.834086 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:50.833920    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5eeacb87-9f34-4314-80f6-d538c5e843fa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5eeacb87-9f34-4314-80f6-d538c5e843fa-workload-certs\") pod \"5eeacb87-9f34-4314-80f6-d538c5e843fa\" (UID: \"5eeacb87-9f34-4314-80f6-d538c5e843fa\") "
Oct 30 22:18:50.834086 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:50.833954    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5eeacb87-9f34-4314-80f6-d538c5e843fa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5eeacb87-9f34-4314-80f6-d538c5e843fa-istio-data\") pod \"5eeacb87-9f34-4314-80f6-d538c5e843fa\" (UID: \"5eeacb87-9f34-4314-80f6-d538c5e843fa\") "
Oct 30 22:18:50.834086 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:50.833998    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5eeacb87-9f34-4314-80f6-d538c5e843fa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5eeacb87-9f34-4314-80f6-d538c5e843fa-istio-envoy\") pod \"5eeacb87-9f34-4314-80f6-d538c5e843fa\" (UID: \"5eeacb87-9f34-4314-80f6-d538c5e843fa\") "
Oct 30 22:18:50.834086 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:50.834061    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5eeacb87-9f34-4314-80f6-d538c5e843fa-credential-socket" pod "5eeacb87-9f34-4314-80f6-d538c5e843fa" (UID: "5eeacb87-9f34-4314-80f6-d538c5e843fa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:18:50.834290 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:50.834109    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5eeacb87-9f34-4314-80f6-d538c5e843fa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eeacb87-9f34-4314-80f6-d538c5e843fa-workload-socket\") pod \"5eeacb87-9f34-4314-80f6-d538c5e843fa\" (UID: \"5eeacb87-9f34-4314-80f6-d538c5e843fa\") "
Oct 30 22:18:50.834290 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:50.834235    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5eeacb87-9f34-4314-80f6-d538c5e843fa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5eeacb87-9f34-4314-80f6-d538c5e843fa-istiod-ca-cert\") pod \"5eeacb87-9f34-4314-80f6-d538c5e843fa\" (UID: \"5eeacb87-9f34-4314-80f6-d538c5e843fa\") "
Oct 30 22:18:50.834290 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:50.834231    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5eeacb87-9f34-4314-80f6-d538c5e843fa-workload-certs" pod "5eeacb87-9f34-4314-80f6-d538c5e843fa" (UID: "5eeacb87-9f34-4314-80f6-d538c5e843fa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:18:50.834443 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:50.834290    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5eeacb87-9f34-4314-80f6-d538c5e843fa-istio-data" pod "5eeacb87-9f34-4314-80f6-d538c5e843fa" (UID: "5eeacb87-9f34-4314-80f6-d538c5e843fa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:18:50.834568 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:50.834461    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5eeacb87-9f34-4314-80f6-d538c5e843fa-workload-socket" pod "5eeacb87-9f34-4314-80f6-d538c5e843fa" (UID: "5eeacb87-9f34-4314-80f6-d538c5e843fa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:18:50.834683 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:50.834662    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5eeacb87-9f34-4314-80f6-d538c5e843fa-istiod-ca-cert" pod "5eeacb87-9f34-4314-80f6-d538c5e843fa" (UID: "5eeacb87-9f34-4314-80f6-d538c5e843fa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:18:50.834985 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:50.834965    5061 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5eeacb87-9f34-4314-80f6-d538c5e843fa-istio-data\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:50.835030 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:50.834995    5061 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eeacb87-9f34-4314-80f6-d538c5e843fa-workload-socket\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:50.835030 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:50.835011    5061 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5eeacb87-9f34-4314-80f6-d538c5e843fa-istiod-ca-cert\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:50.835099 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:50.835028    5061 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eeacb87-9f34-4314-80f6-d538c5e843fa-credential-socket\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:50.835099 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:50.835042    5061 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5eeacb87-9f34-4314-80f6-d538c5e843fa-workload-certs\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:50.836168 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:50.836137    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5eeacb87-9f34-4314-80f6-d538c5e843fa-istio-podinfo" pod "5eeacb87-9f34-4314-80f6-d538c5e843fa" (UID: "5eeacb87-9f34-4314-80f6-d538c5e843fa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:18:50.836313 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:50.836292    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5eeacb87-9f34-4314-80f6-d538c5e843fa-kube-api-access-4z24f" pod "5eeacb87-9f34-4314-80f6-d538c5e843fa" (UID: "5eeacb87-9f34-4314-80f6-d538c5e843fa"). InnerVolumeSpecName "kube-api-access-4z24f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:18:50.836599 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:50.836578    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5eeacb87-9f34-4314-80f6-d538c5e843fa-istio-envoy" pod "5eeacb87-9f34-4314-80f6-d538c5e843fa" (UID: "5eeacb87-9f34-4314-80f6-d538c5e843fa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:18:50.836716 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:50.836597    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5eeacb87-9f34-4314-80f6-d538c5e843fa-istio-token" pod "5eeacb87-9f34-4314-80f6-d538c5e843fa" (UID: "5eeacb87-9f34-4314-80f6-d538c5e843fa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:18:50.936351 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:50.936307    5061 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5eeacb87-9f34-4314-80f6-d538c5e843fa-istio-token\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:50.936351 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:50.936343    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4z24f\" (UniqueName: \"kubernetes.io/projected/5eeacb87-9f34-4314-80f6-d538c5e843fa-kube-api-access-4z24f\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:50.936351 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:50.936358    5061 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5eeacb87-9f34-4314-80f6-d538c5e843fa-istio-podinfo\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:50.936650 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:50.936371    5061 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5eeacb87-9f34-4314-80f6-d538c5e843fa-istio-envoy\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:51.524624 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:51.524588    5061 generic.go:356] "Generic (PLEG): container finished" podID="5eeacb87-9f34-4314-80f6-d538c5e843fa" containerID="14a3d242c861fb7cbe67ebdf86ad5e13a006edd2a1df573e31953065dc78cbbd" exitCode=0
Oct 30 22:18:51.524804 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:51.524635    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-zskrq"
Oct 30 22:18:51.524804 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:51.524668    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-zskrq" event={"ID":"5eeacb87-9f34-4314-80f6-d538c5e843fa","Type":"ContainerDied","Data":"14a3d242c861fb7cbe67ebdf86ad5e13a006edd2a1df573e31953065dc78cbbd"}
Oct 30 22:18:51.524804 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:51.524722    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-zskrq" event={"ID":"5eeacb87-9f34-4314-80f6-d538c5e843fa","Type":"ContainerDied","Data":"416eb4199ede4a69ede54e9ac404afe2e804413f90e9f7a5e47c499473a04741"}
Oct 30 22:18:51.524804 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:51.524787    5061 scope.go:122] "RemoveContainer" containerID="14a3d242c861fb7cbe67ebdf86ad5e13a006edd2a1df573e31953065dc78cbbd"
Oct 30 22:18:51.541617 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:51.541593    5061 scope.go:122] "RemoveContainer" containerID="e57f49eb55bc61870711f23aabe0f47acf4b5e58c65d60588e93417e4590abdf"
Oct 30 22:18:51.552367 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:51.552345    5061 scope.go:122] "RemoveContainer" containerID="14a3d242c861fb7cbe67ebdf86ad5e13a006edd2a1df573e31953065dc78cbbd"
Oct 30 22:18:51.552699 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:18:51.552670    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14a3d242c861fb7cbe67ebdf86ad5e13a006edd2a1df573e31953065dc78cbbd\": container with ID starting with 14a3d242c861fb7cbe67ebdf86ad5e13a006edd2a1df573e31953065dc78cbbd not found: ID does not exist" containerID="14a3d242c861fb7cbe67ebdf86ad5e13a006edd2a1df573e31953065dc78cbbd"
Oct 30 22:18:51.552787 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:51.552699    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14a3d242c861fb7cbe67ebdf86ad5e13a006edd2a1df573e31953065dc78cbbd"} err="failed to get container status \"14a3d242c861fb7cbe67ebdf86ad5e13a006edd2a1df573e31953065dc78cbbd\": rpc error: code = NotFound desc = could not find container \"14a3d242c861fb7cbe67ebdf86ad5e13a006edd2a1df573e31953065dc78cbbd\": container with ID starting with 14a3d242c861fb7cbe67ebdf86ad5e13a006edd2a1df573e31953065dc78cbbd not found: ID does not exist"
Oct 30 22:18:51.552787 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:51.552717    5061 scope.go:122] "RemoveContainer" containerID="e57f49eb55bc61870711f23aabe0f47acf4b5e58c65d60588e93417e4590abdf"
Oct 30 22:18:51.552982 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:18:51.552948    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e57f49eb55bc61870711f23aabe0f47acf4b5e58c65d60588e93417e4590abdf\": container with ID starting with e57f49eb55bc61870711f23aabe0f47acf4b5e58c65d60588e93417e4590abdf not found: ID does not exist" containerID="e57f49eb55bc61870711f23aabe0f47acf4b5e58c65d60588e93417e4590abdf"
Oct 30 22:18:51.553028 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:51.552987    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e57f49eb55bc61870711f23aabe0f47acf4b5e58c65d60588e93417e4590abdf"} err="failed to get container status \"e57f49eb55bc61870711f23aabe0f47acf4b5e58c65d60588e93417e4590abdf\": rpc error: code = NotFound desc = could not find container \"e57f49eb55bc61870711f23aabe0f47acf4b5e58c65d60588e93417e4590abdf\": container with ID starting with e57f49eb55bc61870711f23aabe0f47acf4b5e58c65d60588e93417e4590abdf not found: ID does not exist"
Oct 30 22:18:51.729113 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:51.729077    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-zskrq"]
Oct 30 22:18:51.776172 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:51.776081    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-zskrq"]
Oct 30 22:18:52.271292 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:52.271208    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f776cb-74d8d8c899-hnzj8" podUID="fd36ba7d-191a-4b1d-bbee-7d34510ecdc0" containerName="payload-pre-processing" containerID="cri-o://ca6df9a8af50dd6294921abac5b1e94220fbec69132e2cb7df6549cb2b91afc4" gracePeriod=25
Oct 30 22:18:52.530574 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:52.530460    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5eeacb87-9f34-4314-80f6-d538c5e843fa" path="/var/lib/kubelet/pods/5eeacb87-9f34-4314-80f6-d538c5e843fa/volumes"
Oct 30 22:18:54.669485 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:18:54.669451    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 22:18:55.263390 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:55.263350    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-f776cb/distinct-2-w3-f776cb-kserve-7b4d4cc954-d88pf" podUID="5b184062-7fac-43d4-ba5f-21567df85da3" containerName="main" containerID="cri-o://4bc79af284930e2e37a47baa13155977e23227bd5f18297f0987384964813230" gracePeriod=45
Oct 30 22:18:55.539440 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:55.539413    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-f776cb/distinct-2-w3-f776cb-kserve-7b4d4cc954-d88pf"
Oct 30 22:18:55.553976 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:55.553947    5061 generic.go:356] "Generic (PLEG): container finished" podID="5b184062-7fac-43d4-ba5f-21567df85da3" containerID="4bc79af284930e2e37a47baa13155977e23227bd5f18297f0987384964813230" exitCode=0
Oct 30 22:18:55.553976 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:55.553977    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-f776cb/distinct-2-w3-f776cb-kserve-7b4d4cc954-d88pf"
Oct 30 22:18:55.554122 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:55.553984    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-f776cb/distinct-2-w3-f776cb-kserve-7b4d4cc954-d88pf" event={"ID":"5b184062-7fac-43d4-ba5f-21567df85da3","Type":"ContainerDied","Data":"4bc79af284930e2e37a47baa13155977e23227bd5f18297f0987384964813230"}
Oct 30 22:18:55.554122 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:55.554028    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-f776cb/distinct-2-w3-f776cb-kserve-7b4d4cc954-d88pf" event={"ID":"5b184062-7fac-43d4-ba5f-21567df85da3","Type":"ContainerDied","Data":"2eba29a30db6219883ca4d99f9b4dddaf8388c49dbe112cedecc71e5f98af3f6"}
Oct 30 22:18:55.554122 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:55.554048    5061 scope.go:122] "RemoveContainer" containerID="4bc79af284930e2e37a47baa13155977e23227bd5f18297f0987384964813230"
Oct 30 22:18:55.563488 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:55.563466    5061 scope.go:122] "RemoveContainer" containerID="4bc79af284930e2e37a47baa13155977e23227bd5f18297f0987384964813230"
Oct 30 22:18:55.563768 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:18:55.563747    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bc79af284930e2e37a47baa13155977e23227bd5f18297f0987384964813230\": container with ID starting with 4bc79af284930e2e37a47baa13155977e23227bd5f18297f0987384964813230 not found: ID does not exist" containerID="4bc79af284930e2e37a47baa13155977e23227bd5f18297f0987384964813230"
Oct 30 22:18:55.563835 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:55.563776    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bc79af284930e2e37a47baa13155977e23227bd5f18297f0987384964813230"} err="failed to get container status \"4bc79af284930e2e37a47baa13155977e23227bd5f18297f0987384964813230\": rpc error: code = NotFound desc = could not find container \"4bc79af284930e2e37a47baa13155977e23227bd5f18297f0987384964813230\": container with ID starting with 4bc79af284930e2e37a47baa13155977e23227bd5f18297f0987384964813230 not found: ID does not exist"
Oct 30 22:18:55.598106 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:55.598079    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b184062-7fac-43d4-ba5f-21567df85da3-tls-certs\" (UniqueName: \"kubernetes.io/secret/5b184062-7fac-43d4-ba5f-21567df85da3-tls-certs\") pod \"5b184062-7fac-43d4-ba5f-21567df85da3\" (UID: \"5b184062-7fac-43d4-ba5f-21567df85da3\") "
Oct 30 22:18:55.598198 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:55.598122    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b184062-7fac-43d4-ba5f-21567df85da3-home\" (UniqueName: \"kubernetes.io/empty-dir/5b184062-7fac-43d4-ba5f-21567df85da3-home\") pod \"5b184062-7fac-43d4-ba5f-21567df85da3\" (UID: \"5b184062-7fac-43d4-ba5f-21567df85da3\") "
Oct 30 22:18:55.598251 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:55.598236    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b184062-7fac-43d4-ba5f-21567df85da3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b184062-7fac-43d4-ba5f-21567df85da3-tmp-dir\") pod \"5b184062-7fac-43d4-ba5f-21567df85da3\" (UID: \"5b184062-7fac-43d4-ba5f-21567df85da3\") "
Oct 30 22:18:55.598292 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:55.598272    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b184062-7fac-43d4-ba5f-21567df85da3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b184062-7fac-43d4-ba5f-21567df85da3-model-cache\") pod \"5b184062-7fac-43d4-ba5f-21567df85da3\" (UID: \"5b184062-7fac-43d4-ba5f-21567df85da3\") "
Oct 30 22:18:55.598328 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:55.598318    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b184062-7fac-43d4-ba5f-21567df85da3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b184062-7fac-43d4-ba5f-21567df85da3-dshm\") pod \"5b184062-7fac-43d4-ba5f-21567df85da3\" (UID: \"5b184062-7fac-43d4-ba5f-21567df85da3\") "
Oct 30 22:18:55.598415 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:55.598399    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b184062-7fac-43d4-ba5f-21567df85da3-kube-api-access-ksz9n\" (UniqueName: \"kubernetes.io/projected/5b184062-7fac-43d4-ba5f-21567df85da3-kube-api-access-ksz9n\") pod \"5b184062-7fac-43d4-ba5f-21567df85da3\" (UID: \"5b184062-7fac-43d4-ba5f-21567df85da3\") "
Oct 30 22:18:55.598476 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:55.598431    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b184062-7fac-43d4-ba5f-21567df85da3-home" pod "5b184062-7fac-43d4-ba5f-21567df85da3" (UID: "5b184062-7fac-43d4-ba5f-21567df85da3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:18:55.598566 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:55.598500    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b184062-7fac-43d4-ba5f-21567df85da3-tmp-dir" pod "5b184062-7fac-43d4-ba5f-21567df85da3" (UID: "5b184062-7fac-43d4-ba5f-21567df85da3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:18:55.598566 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:55.598554    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b184062-7fac-43d4-ba5f-21567df85da3-model-cache" pod "5b184062-7fac-43d4-ba5f-21567df85da3" (UID: "5b184062-7fac-43d4-ba5f-21567df85da3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:18:55.599274 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:55.599239    5061 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b184062-7fac-43d4-ba5f-21567df85da3-home\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:55.599274 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:55.599269    5061 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b184062-7fac-43d4-ba5f-21567df85da3-tmp-dir\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:55.599439 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:55.599285    5061 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b184062-7fac-43d4-ba5f-21567df85da3-model-cache\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:55.600634 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:55.600606    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b184062-7fac-43d4-ba5f-21567df85da3-kube-api-access-ksz9n" pod "5b184062-7fac-43d4-ba5f-21567df85da3" (UID: "5b184062-7fac-43d4-ba5f-21567df85da3"). InnerVolumeSpecName "kube-api-access-ksz9n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:18:55.600720 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:55.600645    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b184062-7fac-43d4-ba5f-21567df85da3-dshm" pod "5b184062-7fac-43d4-ba5f-21567df85da3" (UID: "5b184062-7fac-43d4-ba5f-21567df85da3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:18:55.600720 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:55.600685    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b184062-7fac-43d4-ba5f-21567df85da3-tls-certs" pod "5b184062-7fac-43d4-ba5f-21567df85da3" (UID: "5b184062-7fac-43d4-ba5f-21567df85da3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:18:55.700218 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:55.700187    5061 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b184062-7fac-43d4-ba5f-21567df85da3-dshm\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:55.700218 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:55.700214    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ksz9n\" (UniqueName: \"kubernetes.io/projected/5b184062-7fac-43d4-ba5f-21567df85da3-kube-api-access-ksz9n\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:55.700773 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:55.700230    5061 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b184062-7fac-43d4-ba5f-21567df85da3-tls-certs\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:55.974299 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:55.974261    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-f776cb/distinct-2-w3-f776cb-kserve-7b4d4cc954-d88pf"]
Oct 30 22:18:55.982188 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:55.982159    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-f776cb/distinct-2-w3-f776cb-kserve-7b4d4cc954-d88pf"]
Oct 30 22:18:56.529841 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:18:56.529804    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b184062-7fac-43d4-ba5f-21567df85da3" path="/var/lib/kubelet/pods/5b184062-7fac-43d4-ba5f-21567df85da3/volumes"
Oct 30 22:19:00.158719 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:00.158685    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-c9dc10-576f57cf6c-h5z87"]
Oct 30 22:19:00.159149 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:00.158924    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-c9dc10-576f57cf6c-h5z87" podUID="db0dfe9a-a006-41e5-8d87-72eb15fae883" containerName="maas-api" containerID="cri-o://48f23f17e79d55618fa4c03db7cf0097db5466c8e3fdfacc8cd92c111466daee" gracePeriod=30
Oct 30 22:19:00.471648 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:00.471622    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-c9dc10-576f57cf6c-h5z87"
Oct 30 22:19:00.570673 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:00.570644    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db0dfe9a-a006-41e5-8d87-72eb15fae883-metrics-certs\" (UniqueName: \"kubernetes.io/secret/db0dfe9a-a006-41e5-8d87-72eb15fae883-metrics-certs\") pod \"db0dfe9a-a006-41e5-8d87-72eb15fae883\" (UID: \"db0dfe9a-a006-41e5-8d87-72eb15fae883\") "
Oct 30 22:19:00.570858 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:00.570764    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db0dfe9a-a006-41e5-8d87-72eb15fae883-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db0dfe9a-a006-41e5-8d87-72eb15fae883-openshift-service-ca\") pod \"db0dfe9a-a006-41e5-8d87-72eb15fae883\" (UID: \"db0dfe9a-a006-41e5-8d87-72eb15fae883\") "
Oct 30 22:19:00.570858 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:00.570807    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db0dfe9a-a006-41e5-8d87-72eb15fae883-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db0dfe9a-a006-41e5-8d87-72eb15fae883-maas-api-tls\") pod \"db0dfe9a-a006-41e5-8d87-72eb15fae883\" (UID: \"db0dfe9a-a006-41e5-8d87-72eb15fae883\") "
Oct 30 22:19:00.570991 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:00.570932    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db0dfe9a-a006-41e5-8d87-72eb15fae883-kube-api-access-7l6nl\" (UniqueName: \"kubernetes.io/projected/db0dfe9a-a006-41e5-8d87-72eb15fae883-kube-api-access-7l6nl\") pod \"db0dfe9a-a006-41e5-8d87-72eb15fae883\" (UID: \"db0dfe9a-a006-41e5-8d87-72eb15fae883\") "
Oct 30 22:19:00.571181 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:00.571157    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db0dfe9a-a006-41e5-8d87-72eb15fae883-openshift-service-ca" pod "db0dfe9a-a006-41e5-8d87-72eb15fae883" (UID: "db0dfe9a-a006-41e5-8d87-72eb15fae883"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:19:00.571383 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:00.571367    5061 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db0dfe9a-a006-41e5-8d87-72eb15fae883-openshift-service-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:19:00.572872 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:00.572848    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db0dfe9a-a006-41e5-8d87-72eb15fae883-metrics-certs" pod "db0dfe9a-a006-41e5-8d87-72eb15fae883" (UID: "db0dfe9a-a006-41e5-8d87-72eb15fae883"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:19:00.572973 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:00.572951    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db0dfe9a-a006-41e5-8d87-72eb15fae883-maas-api-tls" pod "db0dfe9a-a006-41e5-8d87-72eb15fae883" (UID: "db0dfe9a-a006-41e5-8d87-72eb15fae883"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:19:00.573016 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:00.572999    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db0dfe9a-a006-41e5-8d87-72eb15fae883-kube-api-access-7l6nl" pod "db0dfe9a-a006-41e5-8d87-72eb15fae883" (UID: "db0dfe9a-a006-41e5-8d87-72eb15fae883"). InnerVolumeSpecName "kube-api-access-7l6nl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:19:00.592621 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:00.592577    5061 generic.go:356] "Generic (PLEG): container finished" podID="db0dfe9a-a006-41e5-8d87-72eb15fae883" containerID="48f23f17e79d55618fa4c03db7cf0097db5466c8e3fdfacc8cd92c111466daee" exitCode=0
Oct 30 22:19:00.592621 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:00.592636    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-c9dc10-576f57cf6c-h5z87"
Oct 30 22:19:00.592621 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:00.592838    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-c9dc10-576f57cf6c-h5z87" event={"ID":"db0dfe9a-a006-41e5-8d87-72eb15fae883","Type":"ContainerDied","Data":"48f23f17e79d55618fa4c03db7cf0097db5466c8e3fdfacc8cd92c111466daee"}
Oct 30 22:19:00.593125 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:00.592945    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-c9dc10-576f57cf6c-h5z87" event={"ID":"db0dfe9a-a006-41e5-8d87-72eb15fae883","Type":"ContainerDied","Data":"c7bbae67aa728a1ea44cfe5dd7da481bf4a62ee42e425be5bc717a9f79eec85a"}
Oct 30 22:19:00.593125 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:00.592970    5061 scope.go:122] "RemoveContainer" containerID="48f23f17e79d55618fa4c03db7cf0097db5466c8e3fdfacc8cd92c111466daee"
Oct 30 22:19:00.607521 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:00.607495    5061 scope.go:122] "RemoveContainer" containerID="48f23f17e79d55618fa4c03db7cf0097db5466c8e3fdfacc8cd92c111466daee"
Oct 30 22:19:00.608301 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:19:00.608275    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48f23f17e79d55618fa4c03db7cf0097db5466c8e3fdfacc8cd92c111466daee\": container with ID starting with 48f23f17e79d55618fa4c03db7cf0097db5466c8e3fdfacc8cd92c111466daee not found: ID does not exist" containerID="48f23f17e79d55618fa4c03db7cf0097db5466c8e3fdfacc8cd92c111466daee"
Oct 30 22:19:00.608415 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:00.608310    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48f23f17e79d55618fa4c03db7cf0097db5466c8e3fdfacc8cd92c111466daee"} err="failed to get container status \"48f23f17e79d55618fa4c03db7cf0097db5466c8e3fdfacc8cd92c111466daee\": rpc error: code = NotFound desc = could not find container \"48f23f17e79d55618fa4c03db7cf0097db5466c8e3fdfacc8cd92c111466daee\": container with ID starting with 48f23f17e79d55618fa4c03db7cf0097db5466c8e3fdfacc8cd92c111466daee not found: ID does not exist"
Oct 30 22:19:00.673013 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:00.672989    5061 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db0dfe9a-a006-41e5-8d87-72eb15fae883-metrics-certs\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:19:00.673013 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:00.673011    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db0dfe9a-a006-41e5-8d87-72eb15fae883-maas-api-tls\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:19:00.673180 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:00.673021    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7l6nl\" (UniqueName: \"kubernetes.io/projected/db0dfe9a-a006-41e5-8d87-72eb15fae883-kube-api-access-7l6nl\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:19:00.753212 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:00.753102    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-c9dc10-576f57cf6c-h5z87"]
Oct 30 22:19:00.756609 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:00.756582    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-c9dc10-576f57cf6c-h5z87"]
Oct 30 22:19:00.771441 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:00.771409    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f776cb-74d8d8c899-hnzj8" podUID="fd36ba7d-191a-4b1d-bbee-7d34510ecdc0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:19:02.330831 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:02.330798    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-c9dc10-pcbnw"]
Oct 30 22:19:02.377305 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:02.377271    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-c9dc10-pcbnw"]
Oct 30 22:19:02.530697 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:02.530662    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd7c97c5-1d79-47ea-a22b-eb6b5fd55ab6" path="/var/lib/kubelet/pods/bd7c97c5-1d79-47ea-a22b-eb6b5fd55ab6/volumes"
Oct 30 22:19:02.531151 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:02.531136    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db0dfe9a-a006-41e5-8d87-72eb15fae883" path="/var/lib/kubelet/pods/db0dfe9a-a006-41e5-8d87-72eb15fae883/volumes"
Oct 30 22:19:08.946346 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:08.946257    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91b15223-zw69h"]
Oct 30 22:19:08.947120 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:08.947096    5061 cpu_manager.go:414] "containerMap: removing container" podUID="5eeacb87-9f34-4314-80f6-d538c5e843fa" containerName="istio-proxy"
Oct 30 22:19:08.947223 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:08.947123    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5eeacb87-9f34-4314-80f6-d538c5e843fa" containerName="istio-proxy"
Oct 30 22:19:08.947223 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:08.947158    5061 cpu_manager.go:414] "containerMap: removing container" podUID="5b184062-7fac-43d4-ba5f-21567df85da3" containerName="main"
Oct 30 22:19:08.947223 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:08.947163    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b184062-7fac-43d4-ba5f-21567df85da3" containerName="main"
Oct 30 22:19:08.947223 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:08.947179    5061 cpu_manager.go:414] "containerMap: removing container" podUID="5eeacb87-9f34-4314-80f6-d538c5e843fa" containerName="istio-proxy"
Oct 30 22:19:08.947223 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:08.947184    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5eeacb87-9f34-4314-80f6-d538c5e843fa" containerName="istio-proxy"
Oct 30 22:19:08.947223 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:08.947213    5061 cpu_manager.go:414] "containerMap: removing container" podUID="db0dfe9a-a006-41e5-8d87-72eb15fae883" containerName="maas-api"
Oct 30 22:19:08.947223 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:08.947219    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db0dfe9a-a006-41e5-8d87-72eb15fae883" containerName="maas-api"
Oct 30 22:19:08.947603 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:08.947329    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b184062-7fac-43d4-ba5f-21567df85da3" containerName="main"
Oct 30 22:19:08.947603 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:08.947342    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="5eeacb87-9f34-4314-80f6-d538c5e843fa" containerName="istio-proxy"
Oct 30 22:19:08.947603 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:08.947351    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="db0dfe9a-a006-41e5-8d87-72eb15fae883" containerName="maas-api"
Oct 30 22:19:08.952467 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:08.952447    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91b15223-zw69h"
Oct 30 22:19:09.010661 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:09.010639    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91b15223-zw69h"]
Oct 30 22:19:09.049804 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:09.049779    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:19:09.049994 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:09.049980    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gdmfg\""
Oct 30 22:19:09.074496 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:09.074469    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/305de000-e58b-46d2-b39e-3b7c3abd4bc3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91b15223-zw69h\" (UID: \"305de000-e58b-46d2-b39e-3b7c3abd4bc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91b15223-zw69h"
Oct 30 22:19:09.074625 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:09.074573    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsp2t\" (UniqueName: \"kubernetes.io/projected/305de000-e58b-46d2-b39e-3b7c3abd4bc3-kube-api-access-wsp2t\") pod \"maas-api-revoke-sub-91b15223-zw69h\" (UID: \"305de000-e58b-46d2-b39e-3b7c3abd4bc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91b15223-zw69h"
Oct 30 22:19:09.074713 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:09.074694    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/305de000-e58b-46d2-b39e-3b7c3abd4bc3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91b15223-zw69h\" (UID: \"305de000-e58b-46d2-b39e-3b7c3abd4bc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91b15223-zw69h"
Oct 30 22:19:09.177298 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:09.177263    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/305de000-e58b-46d2-b39e-3b7c3abd4bc3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91b15223-zw69h\" (UID: \"305de000-e58b-46d2-b39e-3b7c3abd4bc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91b15223-zw69h"
Oct 30 22:19:09.177469 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:09.177349    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsp2t\" (UniqueName: \"kubernetes.io/projected/305de000-e58b-46d2-b39e-3b7c3abd4bc3-kube-api-access-wsp2t\") pod \"maas-api-revoke-sub-91b15223-zw69h\" (UID: \"305de000-e58b-46d2-b39e-3b7c3abd4bc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91b15223-zw69h"
Oct 30 22:19:09.177469 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:09.177439    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/305de000-e58b-46d2-b39e-3b7c3abd4bc3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91b15223-zw69h\" (UID: \"305de000-e58b-46d2-b39e-3b7c3abd4bc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91b15223-zw69h"
Oct 30 22:19:09.178234 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:09.178213    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/305de000-e58b-46d2-b39e-3b7c3abd4bc3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91b15223-zw69h\" (UID: \"305de000-e58b-46d2-b39e-3b7c3abd4bc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91b15223-zw69h"
Oct 30 22:19:09.179867 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:09.179847    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/305de000-e58b-46d2-b39e-3b7c3abd4bc3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91b15223-zw69h\" (UID: \"305de000-e58b-46d2-b39e-3b7c3abd4bc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91b15223-zw69h"
Oct 30 22:19:09.232910 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:09.232856    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsp2t\" (UniqueName: \"kubernetes.io/projected/305de000-e58b-46d2-b39e-3b7c3abd4bc3-kube-api-access-wsp2t\") pod \"maas-api-revoke-sub-91b15223-zw69h\" (UID: \"305de000-e58b-46d2-b39e-3b7c3abd4bc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91b15223-zw69h"
Oct 30 22:19:09.268648 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:09.268616    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91b15223-zw69h"
Oct 30 22:19:09.647694 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:09.647577    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91b15223-zw69h"]
Oct 30 22:19:09.652858 ip-10-0-130-60 kubenswrapper[5061]: W1030 22:19:09.652796    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod305de000_e58b_46d2_b39e_3b7c3abd4bc3.slice/crio-ed7006ef0bc9bca055a1446547e6c4da828b3a25f065b1826983ae6c404d51ea WatchSource:0}: Error finding container ed7006ef0bc9bca055a1446547e6c4da828b3a25f065b1826983ae6c404d51ea: Status 404 returned error can't find the container with id ed7006ef0bc9bca055a1446547e6c4da828b3a25f065b1826983ae6c404d51ea
Oct 30 22:19:10.669190 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:10.669154    5061 generic.go:356] "Generic (PLEG): container finished" podID="305de000-e58b-46d2-b39e-3b7c3abd4bc3" containerID="1772ccf2861a0b213f8e446a375c35a4eae685b64fa0d5e62ad46a77423657dc" exitCode=0
Oct 30 22:19:10.669753 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:10.669245    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91b15223-zw69h" event={"ID":"305de000-e58b-46d2-b39e-3b7c3abd4bc3","Type":"ContainerDied","Data":"1772ccf2861a0b213f8e446a375c35a4eae685b64fa0d5e62ad46a77423657dc"}
Oct 30 22:19:10.669753 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:10.669304    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91b15223-zw69h" event={"ID":"305de000-e58b-46d2-b39e-3b7c3abd4bc3","Type":"ContainerStarted","Data":"ed7006ef0bc9bca055a1446547e6c4da828b3a25f065b1826983ae6c404d51ea"}
Oct 30 22:19:10.776246 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:10.774233    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f776cb-74d8d8c899-hnzj8" podUID="fd36ba7d-191a-4b1d-bbee-7d34510ecdc0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:19:11.880908 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:11.880884    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91b15223-zw69h"
Oct 30 22:19:11.963046 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:11.962968    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/305de000-e58b-46d2-b39e-3b7c3abd4bc3-kube-api-access-wsp2t\" (UniqueName: \"kubernetes.io/projected/305de000-e58b-46d2-b39e-3b7c3abd4bc3-kube-api-access-wsp2t\") pod \"305de000-e58b-46d2-b39e-3b7c3abd4bc3\" (UID: \"305de000-e58b-46d2-b39e-3b7c3abd4bc3\") "
Oct 30 22:19:11.963046 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:11.963021    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/305de000-e58b-46d2-b39e-3b7c3abd4bc3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/305de000-e58b-46d2-b39e-3b7c3abd4bc3-maas-api-xks-ca\") pod \"305de000-e58b-46d2-b39e-3b7c3abd4bc3\" (UID: \"305de000-e58b-46d2-b39e-3b7c3abd4bc3\") "
Oct 30 22:19:11.963225 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:11.963066    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/305de000-e58b-46d2-b39e-3b7c3abd4bc3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/305de000-e58b-46d2-b39e-3b7c3abd4bc3-maas-api-openshift-ca\") pod \"305de000-e58b-46d2-b39e-3b7c3abd4bc3\" (UID: \"305de000-e58b-46d2-b39e-3b7c3abd4bc3\") "
Oct 30 22:19:11.963732 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:11.963702    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/305de000-e58b-46d2-b39e-3b7c3abd4bc3-maas-api-openshift-ca" pod "305de000-e58b-46d2-b39e-3b7c3abd4bc3" (UID: "305de000-e58b-46d2-b39e-3b7c3abd4bc3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:19:11.965921 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:11.965895    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/305de000-e58b-46d2-b39e-3b7c3abd4bc3-kube-api-access-wsp2t" pod "305de000-e58b-46d2-b39e-3b7c3abd4bc3" (UID: "305de000-e58b-46d2-b39e-3b7c3abd4bc3"). InnerVolumeSpecName "kube-api-access-wsp2t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:19:11.967280 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:11.967256    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/305de000-e58b-46d2-b39e-3b7c3abd4bc3-maas-api-xks-ca" pod "305de000-e58b-46d2-b39e-3b7c3abd4bc3" (UID: "305de000-e58b-46d2-b39e-3b7c3abd4bc3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:19:12.065374 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:12.065347    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsp2t\" (UniqueName: \"kubernetes.io/projected/305de000-e58b-46d2-b39e-3b7c3abd4bc3-kube-api-access-wsp2t\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:19:12.065519 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:12.065378    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/305de000-e58b-46d2-b39e-3b7c3abd4bc3-maas-api-xks-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:19:12.065519 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:12.065394    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/305de000-e58b-46d2-b39e-3b7c3abd4bc3-maas-api-openshift-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:19:12.515568 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:12.515541    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f776cb-74d8d8c899-hnzj8"
Oct 30 22:19:12.569681 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:12.569646    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd36ba7d-191a-4b1d-bbee-7d34510ecdc0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd36ba7d-191a-4b1d-bbee-7d34510ecdc0-plugins-config-volume\") pod \"fd36ba7d-191a-4b1d-bbee-7d34510ecdc0\" (UID: \"fd36ba7d-191a-4b1d-bbee-7d34510ecdc0\") "
Oct 30 22:19:12.569889 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:12.569754    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd36ba7d-191a-4b1d-bbee-7d34510ecdc0-kube-api-access-j8hc9\" (UniqueName: \"kubernetes.io/projected/fd36ba7d-191a-4b1d-bbee-7d34510ecdc0-kube-api-access-j8hc9\") pod \"fd36ba7d-191a-4b1d-bbee-7d34510ecdc0\" (UID: \"fd36ba7d-191a-4b1d-bbee-7d34510ecdc0\") "
Oct 30 22:19:12.570115 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:12.570083    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd36ba7d-191a-4b1d-bbee-7d34510ecdc0-plugins-config-volume" pod "fd36ba7d-191a-4b1d-bbee-7d34510ecdc0" (UID: "fd36ba7d-191a-4b1d-bbee-7d34510ecdc0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:19:12.570509 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:12.570473    5061 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd36ba7d-191a-4b1d-bbee-7d34510ecdc0-plugins-config-volume\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:19:12.572118 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:12.572090    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd36ba7d-191a-4b1d-bbee-7d34510ecdc0-kube-api-access-j8hc9" pod "fd36ba7d-191a-4b1d-bbee-7d34510ecdc0" (UID: "fd36ba7d-191a-4b1d-bbee-7d34510ecdc0"). InnerVolumeSpecName "kube-api-access-j8hc9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:19:12.672100 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:12.672056    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8hc9\" (UniqueName: \"kubernetes.io/projected/fd36ba7d-191a-4b1d-bbee-7d34510ecdc0-kube-api-access-j8hc9\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:19:12.692745 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:12.692716    5061 generic.go:356] "Generic (PLEG): container finished" podID="fd36ba7d-191a-4b1d-bbee-7d34510ecdc0" containerID="ca6df9a8af50dd6294921abac5b1e94220fbec69132e2cb7df6549cb2b91afc4" exitCode=0
Oct 30 22:19:12.692885 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:12.692758    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f776cb-74d8d8c899-hnzj8"
Oct 30 22:19:12.692885 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:12.692797    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f776cb-74d8d8c899-hnzj8" event={"ID":"fd36ba7d-191a-4b1d-bbee-7d34510ecdc0","Type":"ContainerDied","Data":"ca6df9a8af50dd6294921abac5b1e94220fbec69132e2cb7df6549cb2b91afc4"}
Oct 30 22:19:12.692885 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:12.692855    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f776cb-74d8d8c899-hnzj8" event={"ID":"fd36ba7d-191a-4b1d-bbee-7d34510ecdc0","Type":"ContainerDied","Data":"d644bea2ed2074747c211130c5dd102eac419ba934bba8fb4636b16ff08f9ba8"}
Oct 30 22:19:12.692885 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:12.692878    5061 scope.go:122] "RemoveContainer" containerID="ca6df9a8af50dd6294921abac5b1e94220fbec69132e2cb7df6549cb2b91afc4"
Oct 30 22:19:12.694480 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:12.694453    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91b15223-zw69h" event={"ID":"305de000-e58b-46d2-b39e-3b7c3abd4bc3","Type":"ContainerDied","Data":"ed7006ef0bc9bca055a1446547e6c4da828b3a25f065b1826983ae6c404d51ea"}
Oct 30 22:19:12.694480 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:12.694472    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91b15223-zw69h"
Oct 30 22:19:12.694614 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:12.694493    5061 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ed7006ef0bc9bca055a1446547e6c4da828b3a25f065b1826983ae6c404d51ea"
Oct 30 22:19:12.703768 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:12.703752    5061 scope.go:122] "RemoveContainer" containerID="ca6df9a8af50dd6294921abac5b1e94220fbec69132e2cb7df6549cb2b91afc4"
Oct 30 22:19:12.704021 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:19:12.704002    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca6df9a8af50dd6294921abac5b1e94220fbec69132e2cb7df6549cb2b91afc4\": container with ID starting with ca6df9a8af50dd6294921abac5b1e94220fbec69132e2cb7df6549cb2b91afc4 not found: ID does not exist" containerID="ca6df9a8af50dd6294921abac5b1e94220fbec69132e2cb7df6549cb2b91afc4"
Oct 30 22:19:12.704068 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:12.704028    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca6df9a8af50dd6294921abac5b1e94220fbec69132e2cb7df6549cb2b91afc4"} err="failed to get container status \"ca6df9a8af50dd6294921abac5b1e94220fbec69132e2cb7df6549cb2b91afc4\": rpc error: code = NotFound desc = could not find container \"ca6df9a8af50dd6294921abac5b1e94220fbec69132e2cb7df6549cb2b91afc4\": container with ID starting with ca6df9a8af50dd6294921abac5b1e94220fbec69132e2cb7df6549cb2b91afc4 not found: ID does not exist"
Oct 30 22:19:12.898651 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:12.898620    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-f776cb-74d8d8c899-hnzj8"]
Oct 30 22:19:12.901827 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:12.901800    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-f776cb-74d8d8c899-hnzj8"]
Oct 30 22:19:14.529749 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:14.529717    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd36ba7d-191a-4b1d-bbee-7d34510ecdc0" path="/var/lib/kubelet/pods/fd36ba7d-191a-4b1d-bbee-7d34510ecdc0/volumes"
Oct 30 22:19:24.667714 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:19:24.667664    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 22:19:28.579353 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:28.579320    5061 scope.go:122] "RemoveContainer" containerID="3b52d61ffc4eb7b8d1a162566939e46bf8e201a1d92bbe5360b854d6e1080245"
Oct 30 22:19:44.650804 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:44.650770    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dae567fc-z9w97"]
Oct 30 22:19:44.651602 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:44.651580    5061 cpu_manager.go:414] "containerMap: removing container" podUID="fd36ba7d-191a-4b1d-bbee-7d34510ecdc0" containerName="payload-pre-processing"
Oct 30 22:19:44.651602 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:44.651601    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd36ba7d-191a-4b1d-bbee-7d34510ecdc0" containerName="payload-pre-processing"
Oct 30 22:19:44.651847 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:44.651661    5061 cpu_manager.go:414] "containerMap: removing container" podUID="305de000-e58b-46d2-b39e-3b7c3abd4bc3" containerName="revoke-keys"
Oct 30 22:19:44.651847 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:44.651669    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="305de000-e58b-46d2-b39e-3b7c3abd4bc3" containerName="revoke-keys"
Oct 30 22:19:44.651847 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:44.651819    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd36ba7d-191a-4b1d-bbee-7d34510ecdc0" containerName="payload-pre-processing"
Oct 30 22:19:44.651847 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:44.651835    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="5eeacb87-9f34-4314-80f6-d538c5e843fa" containerName="istio-proxy"
Oct 30 22:19:44.652053 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:44.651853    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="305de000-e58b-46d2-b39e-3b7c3abd4bc3" containerName="revoke-keys"
Oct 30 22:19:44.656972 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:44.656952    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dae567fc-z9w97"
Oct 30 22:19:44.701971 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:44.701947    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91211992-5vwmg"]
Oct 30 22:19:44.706265 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:44.706246    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91211992-5vwmg"
Oct 30 22:19:44.708490 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:44.708467    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:19:44.728377 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:44.728349    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f45e5e6c-e786-4b83-ab69-a25594e59764-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dae567fc-z9w97\" (UID: \"f45e5e6c-e786-4b83-ab69-a25594e59764\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dae567fc-z9w97"
Oct 30 22:19:44.728479 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:44.728447    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nh9rf\" (UniqueName: \"kubernetes.io/projected/40ac0057-5b20-4af0-89ea-037f7400146a-kube-api-access-nh9rf\") pod \"maas-api-revoke-sub-91211992-5vwmg\" (UID: \"40ac0057-5b20-4af0-89ea-037f7400146a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91211992-5vwmg"
Oct 30 22:19:44.728557 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:44.728493    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gh8cr\" (UniqueName: \"kubernetes.io/projected/f45e5e6c-e786-4b83-ab69-a25594e59764-kube-api-access-gh8cr\") pod \"maas-api-revoke-sub-dae567fc-z9w97\" (UID: \"f45e5e6c-e786-4b83-ab69-a25594e59764\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dae567fc-z9w97"
Oct 30 22:19:44.728679 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:44.728655    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f45e5e6c-e786-4b83-ab69-a25594e59764-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dae567fc-z9w97\" (UID: \"f45e5e6c-e786-4b83-ab69-a25594e59764\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dae567fc-z9w97"
Oct 30 22:19:44.728744 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:44.728703    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40ac0057-5b20-4af0-89ea-037f7400146a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91211992-5vwmg\" (UID: \"40ac0057-5b20-4af0-89ea-037f7400146a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91211992-5vwmg"
Oct 30 22:19:44.728868 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:44.728843    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40ac0057-5b20-4af0-89ea-037f7400146a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91211992-5vwmg\" (UID: \"40ac0057-5b20-4af0-89ea-037f7400146a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91211992-5vwmg"
Oct 30 22:19:44.745466 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:44.745445    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gdmfg\""
Oct 30 22:19:44.759005 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:44.758982    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dae567fc-z9w97"]
Oct 30 22:19:44.763332 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:44.763312    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91211992-5vwmg"]
Oct 30 22:19:44.829620 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:44.829586    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f45e5e6c-e786-4b83-ab69-a25594e59764-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dae567fc-z9w97\" (UID: \"f45e5e6c-e786-4b83-ab69-a25594e59764\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dae567fc-z9w97"
Oct 30 22:19:44.829795 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:44.829630    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40ac0057-5b20-4af0-89ea-037f7400146a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91211992-5vwmg\" (UID: \"40ac0057-5b20-4af0-89ea-037f7400146a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91211992-5vwmg"
Oct 30 22:19:44.829795 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:44.829700    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40ac0057-5b20-4af0-89ea-037f7400146a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91211992-5vwmg\" (UID: \"40ac0057-5b20-4af0-89ea-037f7400146a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91211992-5vwmg"
Oct 30 22:19:44.829925 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:44.829829    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f45e5e6c-e786-4b83-ab69-a25594e59764-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dae567fc-z9w97\" (UID: \"f45e5e6c-e786-4b83-ab69-a25594e59764\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dae567fc-z9w97"
Oct 30 22:19:44.829925 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:44.829910    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nh9rf\" (UniqueName: \"kubernetes.io/projected/40ac0057-5b20-4af0-89ea-037f7400146a-kube-api-access-nh9rf\") pod \"maas-api-revoke-sub-91211992-5vwmg\" (UID: \"40ac0057-5b20-4af0-89ea-037f7400146a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91211992-5vwmg"
Oct 30 22:19:44.830030 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:44.829977    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gh8cr\" (UniqueName: \"kubernetes.io/projected/f45e5e6c-e786-4b83-ab69-a25594e59764-kube-api-access-gh8cr\") pod \"maas-api-revoke-sub-dae567fc-z9w97\" (UID: \"f45e5e6c-e786-4b83-ab69-a25594e59764\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dae567fc-z9w97"
Oct 30 22:19:44.830399 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:44.830365    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f45e5e6c-e786-4b83-ab69-a25594e59764-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dae567fc-z9w97\" (UID: \"f45e5e6c-e786-4b83-ab69-a25594e59764\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dae567fc-z9w97"
Oct 30 22:19:44.830564 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:44.830411    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40ac0057-5b20-4af0-89ea-037f7400146a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91211992-5vwmg\" (UID: \"40ac0057-5b20-4af0-89ea-037f7400146a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91211992-5vwmg"
Oct 30 22:19:44.832177 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:44.832156    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40ac0057-5b20-4af0-89ea-037f7400146a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91211992-5vwmg\" (UID: \"40ac0057-5b20-4af0-89ea-037f7400146a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91211992-5vwmg"
Oct 30 22:19:44.832258 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:44.832189    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f45e5e6c-e786-4b83-ab69-a25594e59764-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dae567fc-z9w97\" (UID: \"f45e5e6c-e786-4b83-ab69-a25594e59764\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dae567fc-z9w97"
Oct 30 22:19:44.884285 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:44.884250    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nh9rf\" (UniqueName: \"kubernetes.io/projected/40ac0057-5b20-4af0-89ea-037f7400146a-kube-api-access-nh9rf\") pod \"maas-api-revoke-sub-91211992-5vwmg\" (UID: \"40ac0057-5b20-4af0-89ea-037f7400146a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91211992-5vwmg"
Oct 30 22:19:44.884396 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:44.884250    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gh8cr\" (UniqueName: \"kubernetes.io/projected/f45e5e6c-e786-4b83-ab69-a25594e59764-kube-api-access-gh8cr\") pod \"maas-api-revoke-sub-dae567fc-z9w97\" (UID: \"f45e5e6c-e786-4b83-ab69-a25594e59764\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dae567fc-z9w97"
Oct 30 22:19:44.970050 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:44.969982    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dae567fc-z9w97"
Oct 30 22:19:45.022517 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:45.022490    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91211992-5vwmg"
Oct 30 22:19:45.350980 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:45.350951    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dae567fc-z9w97"]
Oct 30 22:19:45.351394 ip-10-0-130-60 kubenswrapper[5061]: W1030 22:19:45.351346    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf45e5e6c_e786_4b83_ab69_a25594e59764.slice/crio-934afc52acb8fc199621f5c4c876d8c4c7b6592903e8317c48ba0518542c65b3 WatchSource:0}: Error finding container 934afc52acb8fc199621f5c4c876d8c4c7b6592903e8317c48ba0518542c65b3: Status 404 returned error can't find the container with id 934afc52acb8fc199621f5c4c876d8c4c7b6592903e8317c48ba0518542c65b3
Oct 30 22:19:45.472627 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:45.470782    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91211992-5vwmg"]
Oct 30 22:19:45.475152 ip-10-0-130-60 kubenswrapper[5061]: W1030 22:19:45.475113    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40ac0057_5b20_4af0_89ea_037f7400146a.slice/crio-1cc652c15724cdeb2f0089fbacdcc8115cb2ed5abe2fd9cb3cf8f1c58b13b66a WatchSource:0}: Error finding container 1cc652c15724cdeb2f0089fbacdcc8115cb2ed5abe2fd9cb3cf8f1c58b13b66a: Status 404 returned error can't find the container with id 1cc652c15724cdeb2f0089fbacdcc8115cb2ed5abe2fd9cb3cf8f1c58b13b66a
Oct 30 22:19:45.936383 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:45.936291    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dae567fc-z9w97" event={"ID":"f45e5e6c-e786-4b83-ab69-a25594e59764","Type":"ContainerStarted","Data":"105e25a4dfd53227a13fdb06989c601395d72f957f744fabdbe7f62be6c46283"}
Oct 30 22:19:45.936383 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:45.936350    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dae567fc-z9w97" event={"ID":"f45e5e6c-e786-4b83-ab69-a25594e59764","Type":"ContainerStarted","Data":"934afc52acb8fc199621f5c4c876d8c4c7b6592903e8317c48ba0518542c65b3"}
Oct 30 22:19:45.938125 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:45.938098    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91211992-5vwmg" event={"ID":"40ac0057-5b20-4af0-89ea-037f7400146a","Type":"ContainerStarted","Data":"f43e71ca1cab8f496e9a514e74a35faa45f1915b1543b00ea13992e68d9a2eea"}
Oct 30 22:19:45.938235 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:45.938146    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91211992-5vwmg" event={"ID":"40ac0057-5b20-4af0-89ea-037f7400146a","Type":"ContainerStarted","Data":"1cc652c15724cdeb2f0089fbacdcc8115cb2ed5abe2fd9cb3cf8f1c58b13b66a"}
Oct 30 22:19:46.046265 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:46.046219    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dae567fc-z9w97" podStartSLOduration=2.0462064 podStartE2EDuration="2.0462064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:19:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:19:46.045470704 +0000 UTC m=+2239.658240479" watchObservedRunningTime="2026-10-30 22:19:46.0462064 +0000 UTC m=+2239.658976157"
Oct 30 22:19:46.155453 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:46.155399    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91211992-5vwmg" podStartSLOduration=2.155387276 podStartE2EDuration="2.155387276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:19:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:19:46.152661936 +0000 UTC m=+2239.765431697" watchObservedRunningTime="2026-10-30 22:19:46.155387276 +0000 UTC m=+2239.768157033"
Oct 30 22:19:46.947448 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:46.947406    5061 generic.go:356] "Generic (PLEG): container finished" podID="f45e5e6c-e786-4b83-ab69-a25594e59764" containerID="105e25a4dfd53227a13fdb06989c601395d72f957f744fabdbe7f62be6c46283" exitCode=0
Oct 30 22:19:46.947911 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:46.947482    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dae567fc-z9w97" event={"ID":"f45e5e6c-e786-4b83-ab69-a25594e59764","Type":"ContainerDied","Data":"105e25a4dfd53227a13fdb06989c601395d72f957f744fabdbe7f62be6c46283"}
Oct 30 22:19:46.949426 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:46.949400    5061 generic.go:356] "Generic (PLEG): container finished" podID="40ac0057-5b20-4af0-89ea-037f7400146a" containerID="f43e71ca1cab8f496e9a514e74a35faa45f1915b1543b00ea13992e68d9a2eea" exitCode=0
Oct 30 22:19:46.949579 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:46.949436    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91211992-5vwmg" event={"ID":"40ac0057-5b20-4af0-89ea-037f7400146a","Type":"ContainerDied","Data":"f43e71ca1cab8f496e9a514e74a35faa45f1915b1543b00ea13992e68d9a2eea"}
Oct 30 22:19:48.149539 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:48.149404    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dae567fc-z9w97"
Oct 30 22:19:48.168897 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:48.168877    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91211992-5vwmg"
Oct 30 22:19:48.171367 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:48.171343    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f45e5e6c-e786-4b83-ab69-a25594e59764-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f45e5e6c-e786-4b83-ab69-a25594e59764-maas-api-xks-ca\") pod \"f45e5e6c-e786-4b83-ab69-a25594e59764\" (UID: \"f45e5e6c-e786-4b83-ab69-a25594e59764\") "
Oct 30 22:19:48.171517 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:48.171501    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f45e5e6c-e786-4b83-ab69-a25594e59764-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f45e5e6c-e786-4b83-ab69-a25594e59764-maas-api-openshift-ca\") pod \"f45e5e6c-e786-4b83-ab69-a25594e59764\" (UID: \"f45e5e6c-e786-4b83-ab69-a25594e59764\") "
Oct 30 22:19:48.171621 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:48.171605    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f45e5e6c-e786-4b83-ab69-a25594e59764-kube-api-access-gh8cr\" (UniqueName: \"kubernetes.io/projected/f45e5e6c-e786-4b83-ab69-a25594e59764-kube-api-access-gh8cr\") pod \"f45e5e6c-e786-4b83-ab69-a25594e59764\" (UID: \"f45e5e6c-e786-4b83-ab69-a25594e59764\") "
Oct 30 22:19:48.171813 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:48.171788    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f45e5e6c-e786-4b83-ab69-a25594e59764-maas-api-openshift-ca" pod "f45e5e6c-e786-4b83-ab69-a25594e59764" (UID: "f45e5e6c-e786-4b83-ab69-a25594e59764"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:19:48.172146 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:48.172127    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f45e5e6c-e786-4b83-ab69-a25594e59764-maas-api-openshift-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:19:48.173500 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:48.173479    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f45e5e6c-e786-4b83-ab69-a25594e59764-maas-api-xks-ca" pod "f45e5e6c-e786-4b83-ab69-a25594e59764" (UID: "f45e5e6c-e786-4b83-ab69-a25594e59764"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:19:48.174019 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:48.173995    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f45e5e6c-e786-4b83-ab69-a25594e59764-kube-api-access-gh8cr" pod "f45e5e6c-e786-4b83-ab69-a25594e59764" (UID: "f45e5e6c-e786-4b83-ab69-a25594e59764"). InnerVolumeSpecName "kube-api-access-gh8cr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:19:48.272684 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:48.272601    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40ac0057-5b20-4af0-89ea-037f7400146a-kube-api-access-nh9rf\" (UniqueName: \"kubernetes.io/projected/40ac0057-5b20-4af0-89ea-037f7400146a-kube-api-access-nh9rf\") pod \"40ac0057-5b20-4af0-89ea-037f7400146a\" (UID: \"40ac0057-5b20-4af0-89ea-037f7400146a\") "
Oct 30 22:19:48.272684 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:48.272640    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40ac0057-5b20-4af0-89ea-037f7400146a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40ac0057-5b20-4af0-89ea-037f7400146a-maas-api-openshift-ca\") pod \"40ac0057-5b20-4af0-89ea-037f7400146a\" (UID: \"40ac0057-5b20-4af0-89ea-037f7400146a\") "
Oct 30 22:19:48.272914 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:48.272739    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40ac0057-5b20-4af0-89ea-037f7400146a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40ac0057-5b20-4af0-89ea-037f7400146a-maas-api-xks-ca\") pod \"40ac0057-5b20-4af0-89ea-037f7400146a\" (UID: \"40ac0057-5b20-4af0-89ea-037f7400146a\") "
Oct 30 22:19:48.273064 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:48.273034    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40ac0057-5b20-4af0-89ea-037f7400146a-maas-api-openshift-ca" pod "40ac0057-5b20-4af0-89ea-037f7400146a" (UID: "40ac0057-5b20-4af0-89ea-037f7400146a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:19:48.273300 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:48.273280    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40ac0057-5b20-4af0-89ea-037f7400146a-maas-api-openshift-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:19:48.273396 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:48.273303    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gh8cr\" (UniqueName: \"kubernetes.io/projected/f45e5e6c-e786-4b83-ab69-a25594e59764-kube-api-access-gh8cr\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:19:48.273396 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:48.273315    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f45e5e6c-e786-4b83-ab69-a25594e59764-maas-api-xks-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:19:48.275102 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:48.275068    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40ac0057-5b20-4af0-89ea-037f7400146a-maas-api-xks-ca" pod "40ac0057-5b20-4af0-89ea-037f7400146a" (UID: "40ac0057-5b20-4af0-89ea-037f7400146a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:19:48.275584 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:48.275557    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40ac0057-5b20-4af0-89ea-037f7400146a-kube-api-access-nh9rf" pod "40ac0057-5b20-4af0-89ea-037f7400146a" (UID: "40ac0057-5b20-4af0-89ea-037f7400146a"). InnerVolumeSpecName "kube-api-access-nh9rf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:19:48.374546 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:48.374455    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nh9rf\" (UniqueName: \"kubernetes.io/projected/40ac0057-5b20-4af0-89ea-037f7400146a-kube-api-access-nh9rf\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:19:48.374546 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:48.374542    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40ac0057-5b20-4af0-89ea-037f7400146a-maas-api-xks-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:19:48.969256 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:48.969220    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91211992-5vwmg" event={"ID":"40ac0057-5b20-4af0-89ea-037f7400146a","Type":"ContainerDied","Data":"1cc652c15724cdeb2f0089fbacdcc8115cb2ed5abe2fd9cb3cf8f1c58b13b66a"}
Oct 30 22:19:48.969429 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:48.969281    5061 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1cc652c15724cdeb2f0089fbacdcc8115cb2ed5abe2fd9cb3cf8f1c58b13b66a"
Oct 30 22:19:48.969429 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:48.969344    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91211992-5vwmg"
Oct 30 22:19:48.971041 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:48.971008    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dae567fc-z9w97" event={"ID":"f45e5e6c-e786-4b83-ab69-a25594e59764","Type":"ContainerDied","Data":"934afc52acb8fc199621f5c4c876d8c4c7b6592903e8317c48ba0518542c65b3"}
Oct 30 22:19:48.971184 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:48.971059    5061 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="934afc52acb8fc199621f5c4c876d8c4c7b6592903e8317c48ba0518542c65b3"
Oct 30 22:19:48.971184 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:48.971060    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dae567fc-z9w97"
Oct 30 22:19:52.069353 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:52.069316    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-c1f64a-openshift-default-7d9c46ff4f-pnrz7"]
Oct 30 22:19:52.069939 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:52.069593    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-c1f64a-openshift-default-7d9c46ff4f-pnrz7" podUID="2cf14933-e261-4155-8a97-1a1a707c3550" containerName="istio-proxy" containerID="cri-o://1f5800271acd94f8cdebe0c248c72e85a28f1512066737f0054574e478103aef" gracePeriod=30
Oct 30 22:19:54.666991 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:19:54.666956    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 22:19:57.338374 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:57.338304    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c1f64a-openshift-default-7d9c46ff4f-pnrz7"
Oct 30 22:19:57.491901 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:57.491869    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cf14933-e261-4155-8a97-1a1a707c3550-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cf14933-e261-4155-8a97-1a1a707c3550-credential-socket\") pod \"2cf14933-e261-4155-8a97-1a1a707c3550\" (UID: \"2cf14933-e261-4155-8a97-1a1a707c3550\") "
Oct 30 22:19:57.492084 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:57.491928    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cf14933-e261-4155-8a97-1a1a707c3550-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cf14933-e261-4155-8a97-1a1a707c3550-workload-certs\") pod \"2cf14933-e261-4155-8a97-1a1a707c3550\" (UID: \"2cf14933-e261-4155-8a97-1a1a707c3550\") "
Oct 30 22:19:57.492084 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:57.492039    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2cf14933-e261-4155-8a97-1a1a707c3550-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cf14933-e261-4155-8a97-1a1a707c3550-istio-podinfo\") pod \"2cf14933-e261-4155-8a97-1a1a707c3550\" (UID: \"2cf14933-e261-4155-8a97-1a1a707c3550\") "
Oct 30 22:19:57.492213 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:57.492116    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cf14933-e261-4155-8a97-1a1a707c3550-kube-api-access-zz5h6\" (UniqueName: \"kubernetes.io/projected/2cf14933-e261-4155-8a97-1a1a707c3550-kube-api-access-zz5h6\") pod \"2cf14933-e261-4155-8a97-1a1a707c3550\" (UID: \"2cf14933-e261-4155-8a97-1a1a707c3550\") "
Oct 30 22:19:57.492213 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:57.492154    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cf14933-e261-4155-8a97-1a1a707c3550-credential-socket" pod "2cf14933-e261-4155-8a97-1a1a707c3550" (UID: "2cf14933-e261-4155-8a97-1a1a707c3550"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:19:57.492213 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:57.492195    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cf14933-e261-4155-8a97-1a1a707c3550-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cf14933-e261-4155-8a97-1a1a707c3550-istio-envoy\") pod \"2cf14933-e261-4155-8a97-1a1a707c3550\" (UID: \"2cf14933-e261-4155-8a97-1a1a707c3550\") "
Oct 30 22:19:57.492379 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:57.492232    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cf14933-e261-4155-8a97-1a1a707c3550-istio-token\" (UniqueName: \"kubernetes.io/projected/2cf14933-e261-4155-8a97-1a1a707c3550-istio-token\") pod \"2cf14933-e261-4155-8a97-1a1a707c3550\" (UID: \"2cf14933-e261-4155-8a97-1a1a707c3550\") "
Oct 30 22:19:57.492379 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:57.492265    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2cf14933-e261-4155-8a97-1a1a707c3550-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cf14933-e261-4155-8a97-1a1a707c3550-istiod-ca-cert\") pod \"2cf14933-e261-4155-8a97-1a1a707c3550\" (UID: \"2cf14933-e261-4155-8a97-1a1a707c3550\") "
Oct 30 22:19:57.492379 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:57.492286    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cf14933-e261-4155-8a97-1a1a707c3550-workload-certs" pod "2cf14933-e261-4155-8a97-1a1a707c3550" (UID: "2cf14933-e261-4155-8a97-1a1a707c3550"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:19:57.492379 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:57.492329    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cf14933-e261-4155-8a97-1a1a707c3550-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cf14933-e261-4155-8a97-1a1a707c3550-workload-socket\") pod \"2cf14933-e261-4155-8a97-1a1a707c3550\" (UID: \"2cf14933-e261-4155-8a97-1a1a707c3550\") "
Oct 30 22:19:57.492619 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:57.492407    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cf14933-e261-4155-8a97-1a1a707c3550-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cf14933-e261-4155-8a97-1a1a707c3550-istio-data\") pod \"2cf14933-e261-4155-8a97-1a1a707c3550\" (UID: \"2cf14933-e261-4155-8a97-1a1a707c3550\") "
Oct 30 22:19:57.492702 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:57.492670    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cf14933-e261-4155-8a97-1a1a707c3550-workload-socket" pod "2cf14933-e261-4155-8a97-1a1a707c3550" (UID: "2cf14933-e261-4155-8a97-1a1a707c3550"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:19:57.492702 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:57.492687    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cf14933-e261-4155-8a97-1a1a707c3550-istiod-ca-cert" pod "2cf14933-e261-4155-8a97-1a1a707c3550" (UID: "2cf14933-e261-4155-8a97-1a1a707c3550"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:19:57.492857 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:57.492826    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cf14933-e261-4155-8a97-1a1a707c3550-istio-data" pod "2cf14933-e261-4155-8a97-1a1a707c3550" (UID: "2cf14933-e261-4155-8a97-1a1a707c3550"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:19:57.493049 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:57.493030    5061 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cf14933-e261-4155-8a97-1a1a707c3550-istio-data\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:19:57.493103 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:57.493056    5061 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cf14933-e261-4155-8a97-1a1a707c3550-credential-socket\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:19:57.493103 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:57.493074    5061 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cf14933-e261-4155-8a97-1a1a707c3550-workload-certs\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:19:57.493103 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:57.493089    5061 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cf14933-e261-4155-8a97-1a1a707c3550-istiod-ca-cert\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:19:57.493205 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:57.493103    5061 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cf14933-e261-4155-8a97-1a1a707c3550-workload-socket\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:19:57.494414 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:57.494392    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2cf14933-e261-4155-8a97-1a1a707c3550-istio-podinfo" pod "2cf14933-e261-4155-8a97-1a1a707c3550" (UID: "2cf14933-e261-4155-8a97-1a1a707c3550"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:19:57.494501 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:57.494461    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cf14933-e261-4155-8a97-1a1a707c3550-kube-api-access-zz5h6" pod "2cf14933-e261-4155-8a97-1a1a707c3550" (UID: "2cf14933-e261-4155-8a97-1a1a707c3550"). InnerVolumeSpecName "kube-api-access-zz5h6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:19:57.494755 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:57.494739    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cf14933-e261-4155-8a97-1a1a707c3550-istio-envoy" pod "2cf14933-e261-4155-8a97-1a1a707c3550" (UID: "2cf14933-e261-4155-8a97-1a1a707c3550"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:19:57.495020 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:57.495007    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cf14933-e261-4155-8a97-1a1a707c3550-istio-token" pod "2cf14933-e261-4155-8a97-1a1a707c3550" (UID: "2cf14933-e261-4155-8a97-1a1a707c3550"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:19:57.594686 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:57.594600    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zz5h6\" (UniqueName: \"kubernetes.io/projected/2cf14933-e261-4155-8a97-1a1a707c3550-kube-api-access-zz5h6\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:19:57.594686 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:57.594631    5061 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cf14933-e261-4155-8a97-1a1a707c3550-istio-envoy\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:19:57.594686 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:57.594640    5061 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2cf14933-e261-4155-8a97-1a1a707c3550-istio-token\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:19:57.594686 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:57.594649    5061 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cf14933-e261-4155-8a97-1a1a707c3550-istio-podinfo\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:19:58.045505 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:58.045463    5061 generic.go:356] "Generic (PLEG): container finished" podID="2cf14933-e261-4155-8a97-1a1a707c3550" containerID="1f5800271acd94f8cdebe0c248c72e85a28f1512066737f0054574e478103aef" exitCode=0
Oct 30 22:19:58.045707 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:58.045519    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c1f64a-openshift-default-7d9c46ff4f-pnrz7"
Oct 30 22:19:58.045707 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:58.045515    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c1f64a-openshift-default-7d9c46ff4f-pnrz7" event={"ID":"2cf14933-e261-4155-8a97-1a1a707c3550","Type":"ContainerDied","Data":"1f5800271acd94f8cdebe0c248c72e85a28f1512066737f0054574e478103aef"}
Oct 30 22:19:58.045707 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:58.045657    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c1f64a-openshift-default-7d9c46ff4f-pnrz7" event={"ID":"2cf14933-e261-4155-8a97-1a1a707c3550","Type":"ContainerDied","Data":"aa1bbc06374cf3a65ce5344fff990fa18549ba2ca842815659b48775ba1b3b88"}
Oct 30 22:19:58.045707 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:58.045679    5061 scope.go:122] "RemoveContainer" containerID="1f5800271acd94f8cdebe0c248c72e85a28f1512066737f0054574e478103aef"
Oct 30 22:19:58.056754 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:58.056737    5061 scope.go:122] "RemoveContainer" containerID="aabf3da76eb49cc69fbbc7a4c143e478fb9db6b1c977cd981438852e705206a5"
Oct 30 22:19:58.066557 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:58.066522    5061 scope.go:122] "RemoveContainer" containerID="1f5800271acd94f8cdebe0c248c72e85a28f1512066737f0054574e478103aef"
Oct 30 22:19:58.066811 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:19:58.066795    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f5800271acd94f8cdebe0c248c72e85a28f1512066737f0054574e478103aef\": container with ID starting with 1f5800271acd94f8cdebe0c248c72e85a28f1512066737f0054574e478103aef not found: ID does not exist" containerID="1f5800271acd94f8cdebe0c248c72e85a28f1512066737f0054574e478103aef"
Oct 30 22:19:58.066876 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:58.066818    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f5800271acd94f8cdebe0c248c72e85a28f1512066737f0054574e478103aef"} err="failed to get container status \"1f5800271acd94f8cdebe0c248c72e85a28f1512066737f0054574e478103aef\": rpc error: code = NotFound desc = could not find container \"1f5800271acd94f8cdebe0c248c72e85a28f1512066737f0054574e478103aef\": container with ID starting with 1f5800271acd94f8cdebe0c248c72e85a28f1512066737f0054574e478103aef not found: ID does not exist"
Oct 30 22:19:58.066876 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:58.066835    5061 scope.go:122] "RemoveContainer" containerID="aabf3da76eb49cc69fbbc7a4c143e478fb9db6b1c977cd981438852e705206a5"
Oct 30 22:19:58.067083 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:19:58.067062    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aabf3da76eb49cc69fbbc7a4c143e478fb9db6b1c977cd981438852e705206a5\": container with ID starting with aabf3da76eb49cc69fbbc7a4c143e478fb9db6b1c977cd981438852e705206a5 not found: ID does not exist" containerID="aabf3da76eb49cc69fbbc7a4c143e478fb9db6b1c977cd981438852e705206a5"
Oct 30 22:19:58.067128 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:58.067089    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aabf3da76eb49cc69fbbc7a4c143e478fb9db6b1c977cd981438852e705206a5"} err="failed to get container status \"aabf3da76eb49cc69fbbc7a4c143e478fb9db6b1c977cd981438852e705206a5\": rpc error: code = NotFound desc = could not find container \"aabf3da76eb49cc69fbbc7a4c143e478fb9db6b1c977cd981438852e705206a5\": container with ID starting with aabf3da76eb49cc69fbbc7a4c143e478fb9db6b1c977cd981438852e705206a5 not found: ID does not exist"
Oct 30 22:19:58.202814 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:58.202783    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-c1f64a-openshift-default-7d9c46ff4f-pnrz7"]
Oct 30 22:19:58.250579 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:58.250544    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-c1f64a-openshift-default-7d9c46ff4f-pnrz7"]
Oct 30 22:19:58.528787 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:19:58.528714    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2cf14933-e261-4155-8a97-1a1a707c3550" path="/var/lib/kubelet/pods/2cf14933-e261-4155-8a97-1a1a707c3550/volumes"
Oct 30 22:20:00.506640 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:00.506607    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-zhvck"]
Oct 30 22:20:00.507242 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:00.507227    5061 cpu_manager.go:414] "containerMap: removing container" podUID="f45e5e6c-e786-4b83-ab69-a25594e59764" containerName="revoke-keys"
Oct 30 22:20:00.507313 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:00.507244    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f45e5e6c-e786-4b83-ab69-a25594e59764" containerName="revoke-keys"
Oct 30 22:20:00.507313 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:00.507281    5061 cpu_manager.go:414] "containerMap: removing container" podUID="40ac0057-5b20-4af0-89ea-037f7400146a" containerName="revoke-keys"
Oct 30 22:20:00.507313 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:00.507287    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40ac0057-5b20-4af0-89ea-037f7400146a" containerName="revoke-keys"
Oct 30 22:20:00.507313 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:00.507301    5061 cpu_manager.go:414] "containerMap: removing container" podUID="2cf14933-e261-4155-8a97-1a1a707c3550" containerName="istio-proxy"
Oct 30 22:20:00.507313 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:00.507306    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cf14933-e261-4155-8a97-1a1a707c3550" containerName="istio-proxy"
Oct 30 22:20:00.507491 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:00.507422    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cf14933-e261-4155-8a97-1a1a707c3550" containerName="istio-proxy"
Oct 30 22:20:00.507491 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:00.507434    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="f45e5e6c-e786-4b83-ab69-a25594e59764" containerName="revoke-keys"
Oct 30 22:20:00.507491 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:00.507447    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="40ac0057-5b20-4af0-89ea-037f7400146a" containerName="revoke-keys"
Oct 30 22:20:00.507491 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:00.507455    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cf14933-e261-4155-8a97-1a1a707c3550" containerName="istio-proxy"
Oct 30 22:20:00.512513 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:00.512494    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-zhvck"
Oct 30 22:20:00.554454 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:00.554419    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-nbmc8"]
Oct 30 22:20:00.555048 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:00.555032    5061 cpu_manager.go:414] "containerMap: removing container" podUID="2cf14933-e261-4155-8a97-1a1a707c3550" containerName="istio-proxy"
Oct 30 22:20:00.555112 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:00.555051    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cf14933-e261-4155-8a97-1a1a707c3550" containerName="istio-proxy"
Oct 30 22:20:00.558641 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:00.558623    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-nbmc8"
Oct 30 22:20:00.612931 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:00.612897    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-zhvck"]
Oct 30 22:20:00.634134 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:00.634107    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a77dd560-9aff-4292-b430-fa9c72d8c053-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-nbmc8\" (UID: \"a77dd560-9aff-4292-b430-fa9c72d8c053\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-nbmc8"
Oct 30 22:20:00.634291 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:00.634138    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbba8625-2b59-4055-a0f2-5746421672e2-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-zhvck\" (UID: \"bbba8625-2b59-4055-a0f2-5746421672e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-zhvck"
Oct 30 22:20:00.634291 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:00.634159    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a77dd560-9aff-4292-b430-fa9c72d8c053-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-nbmc8\" (UID: \"a77dd560-9aff-4292-b430-fa9c72d8c053\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-nbmc8"
Oct 30 22:20:00.634291 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:00.634206    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbba8625-2b59-4055-a0f2-5746421672e2-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-zhvck\" (UID: \"bbba8625-2b59-4055-a0f2-5746421672e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-zhvck"
Oct 30 22:20:00.634291 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:00.634251    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a77dd560-9aff-4292-b430-fa9c72d8c053-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-nbmc8\" (UID: \"a77dd560-9aff-4292-b430-fa9c72d8c053\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-nbmc8"
Oct 30 22:20:00.634291 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:00.634284    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbba8625-2b59-4055-a0f2-5746421672e2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-zhvck\" (UID: \"bbba8625-2b59-4055-a0f2-5746421672e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-zhvck"
Oct 30 22:20:00.634579 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:00.634313    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a77dd560-9aff-4292-b430-fa9c72d8c053-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-nbmc8\" (UID: \"a77dd560-9aff-4292-b430-fa9c72d8c053\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-nbmc8"
Oct 30 22:20:00.634579 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:00.634420    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a77dd560-9aff-4292-b430-fa9c72d8c053-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-nbmc8\" (UID: \"a77dd560-9aff-4292-b430-fa9c72d8c053\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-nbmc8"
Oct 30 22:20:00.634579 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:00.634450    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a77dd560-9aff-4292-b430-fa9c72d8c053-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-nbmc8\" (UID: \"a77dd560-9aff-4292-b430-fa9c72d8c053\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-nbmc8"
Oct 30 22:20:00.634579 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:00.634473    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfkmh\" (UniqueName: \"kubernetes.io/projected/a77dd560-9aff-4292-b430-fa9c72d8c053-kube-api-access-jfkmh\") pod \"maas-default-gateway-openshift-default-694db55c87-nbmc8\" (UID: \"a77dd560-9aff-4292-b430-fa9c72d8c053\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-nbmc8"
Oct 30 22:20:00.634579 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:00.634493    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbba8625-2b59-4055-a0f2-5746421672e2-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-zhvck\" (UID: \"bbba8625-2b59-4055-a0f2-5746421672e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-zhvck"
Oct 30 22:20:00.634579 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:00.634514    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbba8625-2b59-4055-a0f2-5746421672e2-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-zhvck\" (UID: \"bbba8625-2b59-4055-a0f2-5746421672e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-zhvck"
Oct 30 22:20:00.634898 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:00.634593    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbba8625-2b59-4055-a0f2-5746421672e2-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-zhvck\" (UID: \"bbba8625-2b59-4055-a0f2-5746421672e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-zhvck"
Oct 30 22:20:00.634898 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:00.634619    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a77dd560-9aff-4292-b430-fa9c72d8c053-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-nbmc8\" (UID: \"a77dd560-9aff-4292-b430-fa9c72d8c053\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-nbmc8"
Oct 30 22:20:00.634898 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:00.634745    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbba8625-2b59-4055-a0f2-5746421672e2-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-zhvck\" (UID: \"bbba8625-2b59-4055-a0f2-5746421672e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-zhvck"
Oct 30 22:20:00.634898 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:00.634818    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbba8625-2b59-4055-a0f2-5746421672e2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-zhvck\" (UID: \"bbba8625-2b59-4055-a0f2-5746421672e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-zhvck"
Oct 30 22:20:00.634898 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:00.634863    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a77dd560-9aff-4292-b430-fa9c72d8c053-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-nbmc8\" (UID: \"a77dd560-9aff-4292-b430-fa9c72d8c053\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-nbmc8"
Oct 30 22:20:00.634898 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:00.634880    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9gfb\" (UniqueName: \"kubernetes.io/projected/bbba8625-2b59-4055-a0f2-5746421672e2-kube-api-access-h9gfb\") pod \"maas-default-gateway-openshift-default-694db55c87-zhvck\" (UID: \"bbba8625-2b59-4055-a0f2-5746421672e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-zhvck"
Oct 30 22:20:00.669467 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:00.669421    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-nbmc8"]
Oct 30 22:20:00.736625 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:00.736578    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbba8625-2b59-4055-a0f2-5746421672e2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-zhvck\" (UID: \"bbba8625-2b59-4055-a0f2-5746421672e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-zhvck"
Oct 30 22:20:00.736811 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:00.736641    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a77dd560-9aff-4292-b430-fa9c72d8c053-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-nbmc8\" (UID: \"a77dd560-9aff-4292-b430-fa9c72d8c053\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-nbmc8"
Oct 30 22:20:00.736811 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:00.736711    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a77dd560-9aff-4292-b430-fa9c72d8c053-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-nbmc8\" (UID: \"a77dd560-9aff-4292-b430-fa9c72d8c053\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-nbmc8"
Oct 30 22:20:00.736811 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:00.736750    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a77dd560-9aff-4292-b430-fa9c72d8c053-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-nbmc8\" (UID: \"a77dd560-9aff-4292-b430-fa9c72d8c053\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-nbmc8"
Oct 30 22:20:00.736811 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:00.736782    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfkmh\" (UniqueName: \"kubernetes.io/projected/a77dd560-9aff-4292-b430-fa9c72d8c053-kube-api-access-jfkmh\") pod \"maas-default-gateway-openshift-default-694db55c87-nbmc8\" (UID: \"a77dd560-9aff-4292-b430-fa9c72d8c053\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-nbmc8"
Oct 30 22:20:00.737072 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:00.736815    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbba8625-2b59-4055-a0f2-5746421672e2-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-zhvck\" (UID: \"bbba8625-2b59-4055-a0f2-5746421672e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-zhvck"
Oct 30 22:20:00.737072 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:00.736845    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbba8625-2b59-4055-a0f2-5746421672e2-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-zhvck\" (UID: \"bbba8625-2b59-4055-a0f2-5746421672e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-zhvck"
Oct 30 22:20:00.737072 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:00.736881    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbba8625-2b59-4055-a0f2-5746421672e2-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-zhvck\" (UID: \"bbba8625-2b59-4055-a0f2-5746421672e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-zhvck"
Oct 30 22:20:00.737072 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:00.736915    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a77dd560-9aff-4292-b430-fa9c72d8c053-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-nbmc8\" (UID: \"a77dd560-9aff-4292-b430-fa9c72d8c053\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-nbmc8"
Oct 30 22:20:00.737510 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:00.737480    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbba8625-2b59-4055-a0f2-5746421672e2-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-zhvck\" (UID: \"bbba8625-2b59-4055-a0f2-5746421672e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-zhvck"
Oct 30 22:20:00.737659 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:00.737569    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbba8625-2b59-4055-a0f2-5746421672e2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-zhvck\" (UID: \"bbba8625-2b59-4055-a0f2-5746421672e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-zhvck"
Oct 30 22:20:00.737659 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:00.737617    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbba8625-2b59-4055-a0f2-5746421672e2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-zhvck\" (UID: \"bbba8625-2b59-4055-a0f2-5746421672e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-zhvck"
Oct 30 22:20:00.737788 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:00.737697    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a77dd560-9aff-4292-b430-fa9c72d8c053-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-nbmc8\" (UID: \"a77dd560-9aff-4292-b430-fa9c72d8c053\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-nbmc8"
Oct 30 22:20:00.737788 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:00.737724    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9gfb\" (UniqueName: \"kubernetes.io/projected/bbba8625-2b59-4055-a0f2-5746421672e2-kube-api-access-h9gfb\") pod \"maas-default-gateway-openshift-default-694db55c87-zhvck\" (UID: \"bbba8625-2b59-4055-a0f2-5746421672e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-zhvck"
Oct 30 22:20:00.737895 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:00.737802    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a77dd560-9aff-4292-b430-fa9c72d8c053-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-nbmc8\" (UID: \"a77dd560-9aff-4292-b430-fa9c72d8c053\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-nbmc8"
Oct 30 22:20:00.737895 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:00.737829    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbba8625-2b59-4055-a0f2-5746421672e2-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-zhvck\" (UID: \"bbba8625-2b59-4055-a0f2-5746421672e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-zhvck"
Oct 30 22:20:00.737895 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:00.737854    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbba8625-2b59-4055-a0f2-5746421672e2-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-zhvck\" (UID: \"bbba8625-2b59-4055-a0f2-5746421672e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-zhvck"
Oct 30 22:20:00.737895 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:00.737859    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a77dd560-9aff-4292-b430-fa9c72d8c053-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-nbmc8\" (UID: \"a77dd560-9aff-4292-b430-fa9c72d8c053\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-nbmc8"
Oct 30 22:20:00.738102 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:00.737912    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a77dd560-9aff-4292-b430-fa9c72d8c053-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-nbmc8\" (UID: \"a77dd560-9aff-4292-b430-fa9c72d8c053\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-nbmc8"
Oct 30 22:20:00.738348 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:00.738317    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a77dd560-9aff-4292-b430-fa9c72d8c053-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-nbmc8\" (UID: \"a77dd560-9aff-4292-b430-fa9c72d8c053\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-nbmc8"
Oct 30 22:20:00.738469 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:00.737918    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbba8625-2b59-4055-a0f2-5746421672e2-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-zhvck\" (UID: \"bbba8625-2b59-4055-a0f2-5746421672e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-zhvck"
Oct 30 22:20:00.738469 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:00.738424    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a77dd560-9aff-4292-b430-fa9c72d8c053-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-nbmc8\" (UID: \"a77dd560-9aff-4292-b430-fa9c72d8c053\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-nbmc8"
Oct 30 22:20:00.738654 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:00.738550    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a77dd560-9aff-4292-b430-fa9c72d8c053-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-nbmc8\" (UID: \"a77dd560-9aff-4292-b430-fa9c72d8c053\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-nbmc8"
Oct 30 22:20:00.738844 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:00.738816    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbba8625-2b59-4055-a0f2-5746421672e2-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-zhvck\" (UID: \"bbba8625-2b59-4055-a0f2-5746421672e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-zhvck"
Oct 30 22:20:00.738976 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:00.738951    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a77dd560-9aff-4292-b430-fa9c72d8c053-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-nbmc8\" (UID: \"a77dd560-9aff-4292-b430-fa9c72d8c053\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-nbmc8"
Oct 30 22:20:00.739164 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:00.739133    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbba8625-2b59-4055-a0f2-5746421672e2-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-zhvck\" (UID: \"bbba8625-2b59-4055-a0f2-5746421672e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-zhvck"
Oct 30 22:20:00.740023 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:00.739935    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a77dd560-9aff-4292-b430-fa9c72d8c053-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-nbmc8\" (UID: \"a77dd560-9aff-4292-b430-fa9c72d8c053\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-nbmc8"
Oct 30 22:20:00.741710 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:00.741446    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbba8625-2b59-4055-a0f2-5746421672e2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-zhvck\" (UID: \"bbba8625-2b59-4055-a0f2-5746421672e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-zhvck"
Oct 30 22:20:00.741710 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:00.741636    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbba8625-2b59-4055-a0f2-5746421672e2-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-zhvck\" (UID: \"bbba8625-2b59-4055-a0f2-5746421672e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-zhvck"
Oct 30 22:20:00.742155 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:00.742050    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a77dd560-9aff-4292-b430-fa9c72d8c053-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-nbmc8\" (UID: \"a77dd560-9aff-4292-b430-fa9c72d8c053\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-nbmc8"
Oct 30 22:20:00.742390 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:00.742260    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbba8625-2b59-4055-a0f2-5746421672e2-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-zhvck\" (UID: \"bbba8625-2b59-4055-a0f2-5746421672e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-zhvck"
Oct 30 22:20:00.743996 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:00.743949    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a77dd560-9aff-4292-b430-fa9c72d8c053-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-nbmc8\" (UID: \"a77dd560-9aff-4292-b430-fa9c72d8c053\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-nbmc8"
Oct 30 22:20:00.792660 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:00.792584    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbba8625-2b59-4055-a0f2-5746421672e2-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-zhvck\" (UID: \"bbba8625-2b59-4055-a0f2-5746421672e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-zhvck"
Oct 30 22:20:00.792660 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:00.792640    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a77dd560-9aff-4292-b430-fa9c72d8c053-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-nbmc8\" (UID: \"a77dd560-9aff-4292-b430-fa9c72d8c053\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-nbmc8"
Oct 30 22:20:00.792989 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:00.792969    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfkmh\" (UniqueName: \"kubernetes.io/projected/a77dd560-9aff-4292-b430-fa9c72d8c053-kube-api-access-jfkmh\") pod \"maas-default-gateway-openshift-default-694db55c87-nbmc8\" (UID: \"a77dd560-9aff-4292-b430-fa9c72d8c053\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-nbmc8"
Oct 30 22:20:00.793037 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:00.792972    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9gfb\" (UniqueName: \"kubernetes.io/projected/bbba8625-2b59-4055-a0f2-5746421672e2-kube-api-access-h9gfb\") pod \"maas-default-gateway-openshift-default-694db55c87-zhvck\" (UID: \"bbba8625-2b59-4055-a0f2-5746421672e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-zhvck"
Oct 30 22:20:00.826747 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:00.826720    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-zhvck"
Oct 30 22:20:00.872727 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:00.872689    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-nbmc8"
Oct 30 22:20:01.248918 ip-10-0-130-60 kubenswrapper[5061]: W1030 22:20:01.248876    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbba8625_2b59_4055_a0f2_5746421672e2.slice/crio-0e6d71eb1667ae849ce1cce091dcaa791a3fc21cdd570d664337141deaa78d6a WatchSource:0}: Error finding container 0e6d71eb1667ae849ce1cce091dcaa791a3fc21cdd570d664337141deaa78d6a: Status 404 returned error can't find the container with id 0e6d71eb1667ae849ce1cce091dcaa791a3fc21cdd570d664337141deaa78d6a
Oct 30 22:20:01.249156 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:01.249132    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-zhvck"]
Oct 30 22:20:01.251474 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:01.251441    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:20:01.251576 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:01.251504    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:20:01.251624 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:01.251564    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:20:01.295727 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:01.295699    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-nbmc8"]
Oct 30 22:20:01.296406 ip-10-0-130-60 kubenswrapper[5061]: W1030 22:20:01.296374    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda77dd560_9aff_4292_b430_fa9c72d8c053.slice/crio-32291b7395288af65fbe476bd1a4190bb66cb16d661c1c01dc862fb4a1f2505c WatchSource:0}: Error finding container 32291b7395288af65fbe476bd1a4190bb66cb16d661c1c01dc862fb4a1f2505c: Status 404 returned error can't find the container with id 32291b7395288af65fbe476bd1a4190bb66cb16d661c1c01dc862fb4a1f2505c
Oct 30 22:20:01.299454 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:01.299418    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:20:01.299604 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:01.299500    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:20:01.299604 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:01.299565    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:20:02.079754 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:02.079706    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-zhvck" event={"ID":"bbba8625-2b59-4055-a0f2-5746421672e2","Type":"ContainerStarted","Data":"a356d357861c378882837d0e3ea0c0599ff147cd3e7a7425a9ca5bf6e6e7a170"}
Oct 30 22:20:02.080269 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:02.079781    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-zhvck" event={"ID":"bbba8625-2b59-4055-a0f2-5746421672e2","Type":"ContainerStarted","Data":"0e6d71eb1667ae849ce1cce091dcaa791a3fc21cdd570d664337141deaa78d6a"}
Oct 30 22:20:02.081852 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:02.081822    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-nbmc8" event={"ID":"a77dd560-9aff-4292-b430-fa9c72d8c053","Type":"ContainerStarted","Data":"d995bf8cef3c878696b15638377b14b5ae9b4a2e2740cf42a5ca1eb1c0d018b4"}
Oct 30 22:20:02.081992 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:02.081874    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-nbmc8" event={"ID":"a77dd560-9aff-4292-b430-fa9c72d8c053","Type":"ContainerStarted","Data":"32291b7395288af65fbe476bd1a4190bb66cb16d661c1c01dc862fb4a1f2505c"}
Oct 30 22:20:02.195166 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:02.195106    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-zhvck" podStartSLOduration=2.195089719 podStartE2EDuration="2.195089719s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:20:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:20:02.192833641 +0000 UTC m=+2255.805603426" watchObservedRunningTime="2026-10-30 22:20:02.195089719 +0000 UTC m=+2255.807859476"
Oct 30 22:20:02.339935 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:02.339762    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-nbmc8" podStartSLOduration=2.33973636 podStartE2EDuration="2.33973636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:20:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:20:02.317694175 +0000 UTC m=+2255.930463926" watchObservedRunningTime="2026-10-30 22:20:02.33973636 +0000 UTC m=+2255.952506119"
Oct 30 22:20:02.850505 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:02.842917    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-zhvck"
Oct 30 22:20:02.859303 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:02.858918    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-zhvck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body=
Oct 30 22:20:02.859303 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:02.858988    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-zhvck" podUID="bbba8625-2b59-4055-a0f2-5746421672e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused"
Oct 30 22:20:02.873506 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:02.873472    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-nbmc8"
Oct 30 22:20:02.874907 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:02.874876    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-nbmc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body=
Oct 30 22:20:02.875034 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:02.874938    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-nbmc8" podUID="a77dd560-9aff-4292-b430-fa9c72d8c053" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused"
Oct 30 22:20:03.828549 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:03.827958    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-zhvck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body=
Oct 30 22:20:03.828549 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:03.828021    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-zhvck" podUID="bbba8625-2b59-4055-a0f2-5746421672e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused"
Oct 30 22:20:03.873420 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:03.873384    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-nbmc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body=
Oct 30 22:20:03.873630 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:03.873439    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-nbmc8" podUID="a77dd560-9aff-4292-b430-fa9c72d8c053" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused"
Oct 30 22:20:04.342894 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:04.342853    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4215ed-ph8sn"]
Oct 30 22:20:04.355926 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:04.355892    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4215ed-ph8sn"
Oct 30 22:20:04.396489 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:04.396459    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4215ed-ph8sn"]
Oct 30 22:20:04.448553 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:04.444166    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:20:04.448553 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:04.444319    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gdmfg\""
Oct 30 22:20:04.495734 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:04.495699    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chq55\" (UniqueName: \"kubernetes.io/projected/40263768-14ff-472a-b17c-35bbf52809a3-kube-api-access-chq55\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-4215ed-ph8sn\" (UID: \"40263768-14ff-472a-b17c-35bbf52809a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4215ed-ph8sn"
Oct 30 22:20:04.495890 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:04.495870    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40263768-14ff-472a-b17c-35bbf52809a3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-4215ed-ph8sn\" (UID: \"40263768-14ff-472a-b17c-35bbf52809a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4215ed-ph8sn"
Oct 30 22:20:04.495983 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:04.495966    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40263768-14ff-472a-b17c-35bbf52809a3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-4215ed-ph8sn\" (UID: \"40263768-14ff-472a-b17c-35bbf52809a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4215ed-ph8sn"
Oct 30 22:20:04.597414 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:04.597322    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chq55\" (UniqueName: \"kubernetes.io/projected/40263768-14ff-472a-b17c-35bbf52809a3-kube-api-access-chq55\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-4215ed-ph8sn\" (UID: \"40263768-14ff-472a-b17c-35bbf52809a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4215ed-ph8sn"
Oct 30 22:20:04.597815 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:04.597786    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40263768-14ff-472a-b17c-35bbf52809a3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-4215ed-ph8sn\" (UID: \"40263768-14ff-472a-b17c-35bbf52809a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4215ed-ph8sn"
Oct 30 22:20:04.598080 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:04.598035    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40263768-14ff-472a-b17c-35bbf52809a3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-4215ed-ph8sn\" (UID: \"40263768-14ff-472a-b17c-35bbf52809a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4215ed-ph8sn"
Oct 30 22:20:04.598409 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:04.598389    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40263768-14ff-472a-b17c-35bbf52809a3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-4215ed-ph8sn\" (UID: \"40263768-14ff-472a-b17c-35bbf52809a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4215ed-ph8sn"
Oct 30 22:20:04.600318 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:04.600296    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40263768-14ff-472a-b17c-35bbf52809a3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-4215ed-ph8sn\" (UID: \"40263768-14ff-472a-b17c-35bbf52809a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4215ed-ph8sn"
Oct 30 22:20:04.667259 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:04.667219    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chq55\" (UniqueName: \"kubernetes.io/projected/40263768-14ff-472a-b17c-35bbf52809a3-kube-api-access-chq55\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-4215ed-ph8sn\" (UID: \"40263768-14ff-472a-b17c-35bbf52809a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4215ed-ph8sn"
Oct 30 22:20:04.673836 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:04.673802    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4215ed-ph8sn"
Oct 30 22:20:04.831556 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:04.830979    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-zhvck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body=
Oct 30 22:20:04.831556 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:04.831044    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-zhvck" podUID="bbba8625-2b59-4055-a0f2-5746421672e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused"
Oct 30 22:20:04.875817 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:04.875508    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-nbmc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body=
Oct 30 22:20:04.875817 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:04.875594    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-nbmc8" podUID="a77dd560-9aff-4292-b430-fa9c72d8c053" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused"
Oct 30 22:20:05.098198 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:05.098165    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4215ed-ph8sn"]
Oct 30 22:20:05.099060 ip-10-0-130-60 kubenswrapper[5061]: W1030 22:20:05.099029    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40263768_14ff_472a_b17c_35bbf52809a3.slice/crio-5074bc15b6f85057d3de632265dfc4ff8b975034a50023ad6652ce7eaf389a76 WatchSource:0}: Error finding container 5074bc15b6f85057d3de632265dfc4ff8b975034a50023ad6652ce7eaf389a76: Status 404 returned error can't find the container with id 5074bc15b6f85057d3de632265dfc4ff8b975034a50023ad6652ce7eaf389a76
Oct 30 22:20:05.827491 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:05.827453    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-zhvck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body=
Oct 30 22:20:05.827778 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:05.827749    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-zhvck" podUID="bbba8625-2b59-4055-a0f2-5746421672e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused"
Oct 30 22:20:05.874352 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:05.874312    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-nbmc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body=
Oct 30 22:20:05.874780 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:05.874370    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-nbmc8" podUID="a77dd560-9aff-4292-b430-fa9c72d8c053" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused"
Oct 30 22:20:06.162382 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:06.154911    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4215ed-ph8sn" event={"ID":"40263768-14ff-472a-b17c-35bbf52809a3","Type":"ContainerStarted","Data":"739603148ec04cb2bcaf124f97772fe53f62a406823867d81527d53336bdbd90"}
Oct 30 22:20:06.162382 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:06.162129    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4215ed-ph8sn" event={"ID":"40263768-14ff-472a-b17c-35bbf52809a3","Type":"ContainerStarted","Data":"5074bc15b6f85057d3de632265dfc4ff8b975034a50023ad6652ce7eaf389a76"}
Oct 30 22:20:06.267515 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:06.267452    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4215ed-ph8sn" podStartSLOduration=2.267437038 podStartE2EDuration="2.267437038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:20:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:20:06.264775116 +0000 UTC m=+2259.877544898" watchObservedRunningTime="2026-10-30 22:20:06.267437038 +0000 UTC m=+2259.880206797"
Oct 30 22:20:06.833549 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:06.832963    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-zhvck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body=
Oct 30 22:20:06.833549 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:06.833040    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-zhvck" podUID="bbba8625-2b59-4055-a0f2-5746421672e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused"
Oct 30 22:20:06.873850 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:06.873818    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-nbmc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body=
Oct 30 22:20:06.874015 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:06.873878    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-nbmc8" podUID="a77dd560-9aff-4292-b430-fa9c72d8c053" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused"
Oct 30 22:20:07.168700 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:07.168614    5061 generic.go:356] "Generic (PLEG): container finished" podID="40263768-14ff-472a-b17c-35bbf52809a3" containerID="739603148ec04cb2bcaf124f97772fe53f62a406823867d81527d53336bdbd90" exitCode=0
Oct 30 22:20:07.169077 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:07.168794    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4215ed-ph8sn" event={"ID":"40263768-14ff-472a-b17c-35bbf52809a3","Type":"ContainerDied","Data":"739603148ec04cb2bcaf124f97772fe53f62a406823867d81527d53336bdbd90"}
Oct 30 22:20:07.827553 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:07.827492    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-zhvck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body=
Oct 30 22:20:07.827751 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:07.827564    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-zhvck" podUID="bbba8625-2b59-4055-a0f2-5746421672e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused"
Oct 30 22:20:07.873466 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:07.873430    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-nbmc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body=
Oct 30 22:20:07.873622 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:07.873497    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-nbmc8" podUID="a77dd560-9aff-4292-b430-fa9c72d8c053" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused"
Oct 30 22:20:08.361254 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:08.361222    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4215ed-ph8sn"
Oct 30 22:20:08.392204 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:08.392169    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40263768-14ff-472a-b17c-35bbf52809a3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40263768-14ff-472a-b17c-35bbf52809a3-maas-api-xks-ca\") pod \"40263768-14ff-472a-b17c-35bbf52809a3\" (UID: \"40263768-14ff-472a-b17c-35bbf52809a3\") "
Oct 30 22:20:08.392363 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:08.392260    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40263768-14ff-472a-b17c-35bbf52809a3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40263768-14ff-472a-b17c-35bbf52809a3-maas-api-openshift-ca\") pod \"40263768-14ff-472a-b17c-35bbf52809a3\" (UID: \"40263768-14ff-472a-b17c-35bbf52809a3\") "
Oct 30 22:20:08.392415 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:08.392389    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40263768-14ff-472a-b17c-35bbf52809a3-kube-api-access-chq55\" (UniqueName: \"kubernetes.io/projected/40263768-14ff-472a-b17c-35bbf52809a3-kube-api-access-chq55\") pod \"40263768-14ff-472a-b17c-35bbf52809a3\" (UID: \"40263768-14ff-472a-b17c-35bbf52809a3\") "
Oct 30 22:20:08.392733 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:08.392692    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40263768-14ff-472a-b17c-35bbf52809a3-maas-api-openshift-ca" pod "40263768-14ff-472a-b17c-35bbf52809a3" (UID: "40263768-14ff-472a-b17c-35bbf52809a3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:20:08.393136 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:08.393110    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40263768-14ff-472a-b17c-35bbf52809a3-maas-api-openshift-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:20:08.394452 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:08.394425    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40263768-14ff-472a-b17c-35bbf52809a3-maas-api-xks-ca" pod "40263768-14ff-472a-b17c-35bbf52809a3" (UID: "40263768-14ff-472a-b17c-35bbf52809a3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:20:08.394549 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:08.394431    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40263768-14ff-472a-b17c-35bbf52809a3-kube-api-access-chq55" pod "40263768-14ff-472a-b17c-35bbf52809a3" (UID: "40263768-14ff-472a-b17c-35bbf52809a3"). InnerVolumeSpecName "kube-api-access-chq55". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:20:08.494463 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:08.494367    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chq55\" (UniqueName: \"kubernetes.io/projected/40263768-14ff-472a-b17c-35bbf52809a3-kube-api-access-chq55\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:20:08.494463 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:08.494407    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40263768-14ff-472a-b17c-35bbf52809a3-maas-api-xks-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:20:08.828042 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:08.828010    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-zhvck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body=
Oct 30 22:20:08.828273 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:08.828062    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-zhvck" podUID="bbba8625-2b59-4055-a0f2-5746421672e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused"
Oct 30 22:20:08.873306 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:08.873268    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-nbmc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body=
Oct 30 22:20:08.873457 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:08.873339    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-nbmc8" podUID="a77dd560-9aff-4292-b430-fa9c72d8c053" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused"
Oct 30 22:20:09.190085 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:09.190001    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4215ed-ph8sn"
Oct 30 22:20:09.190085 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:09.190027    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4215ed-ph8sn" event={"ID":"40263768-14ff-472a-b17c-35bbf52809a3","Type":"ContainerDied","Data":"5074bc15b6f85057d3de632265dfc4ff8b975034a50023ad6652ce7eaf389a76"}
Oct 30 22:20:09.190085 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:09.190082    5061 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5074bc15b6f85057d3de632265dfc4ff8b975034a50023ad6652ce7eaf389a76"
Oct 30 22:20:10.502551 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:10.498737    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-nbmc8"
Oct 30 22:20:10.502551 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:10.500366    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-nbmc8"
Oct 30 22:20:10.504000 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:10.503880    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-nbmc8"
Oct 30 22:20:10.535007 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:10.534972    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-zhvck"
Oct 30 22:20:10.535304 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:10.535268    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-zhvck"
Oct 30 22:20:10.536265 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:10.536245    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-zhvck"
Oct 30 22:20:13.601551 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:13.601172    5061 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-14af5c/premium-w6-14af5c-kserve-6549647f4-f4gwg" podUID="9b42ee01-c60e-462c-a0e0-b04a52014db6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:20:13.635552 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:13.624666    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2d1f91/premium-w0-2d1f91-kserve-6965b558cf-59lgw" podUID="fb84233d-25e0-48bc-8899-c3a3cb3cfeb9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:20:13.639549 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:13.639370    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2d1f91/simulator-w0-2d1f91-kserve-7699765c74-jwtkb" podUID="44b7d6f6-32d8-4ef4-95bc-eef604fa6b5b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:20:13.720153 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:13.720109    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-17d809/distinct-w4-17d809-kserve-6b8f5c48cd-lhrwp" podUID="ef5e8691-42e1-471b-ba7d-90881028eca7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded"
Oct 30 22:20:13.806551 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:13.803039    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-17d809/premium-w4-17d809-kserve-7c9bdbbcbc-5mtmm" podUID="1e0eca81-7db1-482b-8dd6-5b28fb89e010" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:20:13.950604 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:13.948622    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-17d809/simulator-w4-17d809-kserve-78d67b6f4-7hsvb" podUID="e7337cf8-4f59-4afb-9b50-6b6c9fa4a773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:20:13.950604 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:13.948640    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-14af5c/distinct-w6-14af5c-kserve-7c7cb9d69c-hljl9" podUID="1f8d9486-fb2f-491a-a8d7-c23c21bd116a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:20:13.954547 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:13.952641    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a07620/simulator-w1-a07620-kserve-55b44674f4-4z9lk" podUID="efe7c063-fb0e-4679-8a7c-13f2a0abaa74" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:20:13.954547 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:13.953325    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-14af5c/premium-w6-14af5c-kserve-6549647f4-f4gwg" podUID="9b42ee01-c60e-462c-a0e0-b04a52014db6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:20:13.954547 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:13.953971    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5be0d1/premium-w2-5be0d1-kserve-759b6fb995-c6gp4" podUID="933f2470-68de-4768-a05b-6c454f16f5f2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:20:13.962630 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:13.962601    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-blrrb" podUID="cb4b1ed5-2192-4877-8be7-ca82af831af5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:20:13.985989 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:13.985654    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-14af5c/distinct-2-w6-14af5c-kserve-58d554bdb6-r5r6j" podUID="990f4e94-708e-45f3-b2ad-01cd50b3257b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded"
Oct 30 22:20:14.003332 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:14.003290    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-14af5c/simulator-w6-14af5c-kserve-85b8bd96d7-wlcb9" podUID="39766c7d-5f1d-42db-8920-c0321cd12c99" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:20:14.013638 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:14.013594    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5be0d1/distinct-w2-5be0d1-kserve-6b85698c8-sxn9k" podUID="7a2dbfc5-908f-41d1-baff-074c56d2cf16" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 30 22:20:14.106561 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:14.106277    5061 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-5be0d1/distinct-w2-5be0d1-kserve-6b85698c8-sxn9k" podUID="7a2dbfc5-908f-41d1-baff-074c56d2cf16" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:20:24.669349 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:20:24.669302    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 22:20:35.911639 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:35.911602    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-nv6kl"]
Oct 30 22:20:35.912213 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:35.912198    5061 cpu_manager.go:414] "containerMap: removing container" podUID="40263768-14ff-472a-b17c-35bbf52809a3" containerName="revoke-keys"
Oct 30 22:20:35.912258 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:35.912214    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40263768-14ff-472a-b17c-35bbf52809a3" containerName="revoke-keys"
Oct 30 22:20:35.912338 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:35.912329    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="40263768-14ff-472a-b17c-35bbf52809a3" containerName="revoke-keys"
Oct 30 22:20:35.917476 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:35.917457    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-nv6kl"
Oct 30 22:20:36.011134 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:36.011085    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/732b6214-aa8c-45fc-bd06-c0f75a267960-workload-socket\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-nv6kl\" (UID: \"732b6214-aa8c-45fc-bd06-c0f75a267960\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-nv6kl"
Oct 30 22:20:36.011134 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:36.011134    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9kmg\" (UniqueName: \"kubernetes.io/projected/732b6214-aa8c-45fc-bd06-c0f75a267960-kube-api-access-v9kmg\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-nv6kl\" (UID: \"732b6214-aa8c-45fc-bd06-c0f75a267960\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-nv6kl"
Oct 30 22:20:36.011741 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:36.011276    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/732b6214-aa8c-45fc-bd06-c0f75a267960-credential-socket\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-nv6kl\" (UID: \"732b6214-aa8c-45fc-bd06-c0f75a267960\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-nv6kl"
Oct 30 22:20:36.011741 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:36.011334    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/732b6214-aa8c-45fc-bd06-c0f75a267960-istiod-ca-cert\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-nv6kl\" (UID: \"732b6214-aa8c-45fc-bd06-c0f75a267960\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-nv6kl"
Oct 30 22:20:36.011741 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:36.011366    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/732b6214-aa8c-45fc-bd06-c0f75a267960-istio-podinfo\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-nv6kl\" (UID: \"732b6214-aa8c-45fc-bd06-c0f75a267960\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-nv6kl"
Oct 30 22:20:36.011741 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:36.011475    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/732b6214-aa8c-45fc-bd06-c0f75a267960-istio-data\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-nv6kl\" (UID: \"732b6214-aa8c-45fc-bd06-c0f75a267960\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-nv6kl"
Oct 30 22:20:36.011741 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:36.011560    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/732b6214-aa8c-45fc-bd06-c0f75a267960-istio-envoy\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-nv6kl\" (UID: \"732b6214-aa8c-45fc-bd06-c0f75a267960\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-nv6kl"
Oct 30 22:20:36.011741 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:36.011641    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/732b6214-aa8c-45fc-bd06-c0f75a267960-workload-certs\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-nv6kl\" (UID: \"732b6214-aa8c-45fc-bd06-c0f75a267960\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-nv6kl"
Oct 30 22:20:36.011741 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:36.011690    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/732b6214-aa8c-45fc-bd06-c0f75a267960-istio-token\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-nv6kl\" (UID: \"732b6214-aa8c-45fc-bd06-c0f75a267960\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-nv6kl"
Oct 30 22:20:36.112695 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:36.112657    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/732b6214-aa8c-45fc-bd06-c0f75a267960-credential-socket\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-nv6kl\" (UID: \"732b6214-aa8c-45fc-bd06-c0f75a267960\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-nv6kl"
Oct 30 22:20:36.112885 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:36.112706    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/732b6214-aa8c-45fc-bd06-c0f75a267960-istiod-ca-cert\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-nv6kl\" (UID: \"732b6214-aa8c-45fc-bd06-c0f75a267960\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-nv6kl"
Oct 30 22:20:36.112885 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:36.112821    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/732b6214-aa8c-45fc-bd06-c0f75a267960-istio-podinfo\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-nv6kl\" (UID: \"732b6214-aa8c-45fc-bd06-c0f75a267960\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-nv6kl"
Oct 30 22:20:36.113019 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:36.112967    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/732b6214-aa8c-45fc-bd06-c0f75a267960-istio-data\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-nv6kl\" (UID: \"732b6214-aa8c-45fc-bd06-c0f75a267960\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-nv6kl"
Oct 30 22:20:36.113076 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:36.113045    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/732b6214-aa8c-45fc-bd06-c0f75a267960-istio-envoy\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-nv6kl\" (UID: \"732b6214-aa8c-45fc-bd06-c0f75a267960\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-nv6kl"
Oct 30 22:20:36.113130 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:36.113080    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/732b6214-aa8c-45fc-bd06-c0f75a267960-credential-socket\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-nv6kl\" (UID: \"732b6214-aa8c-45fc-bd06-c0f75a267960\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-nv6kl"
Oct 30 22:20:36.113182 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:36.113149    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/732b6214-aa8c-45fc-bd06-c0f75a267960-workload-certs\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-nv6kl\" (UID: \"732b6214-aa8c-45fc-bd06-c0f75a267960\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-nv6kl"
Oct 30 22:20:36.113248 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:36.113204    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/732b6214-aa8c-45fc-bd06-c0f75a267960-istio-token\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-nv6kl\" (UID: \"732b6214-aa8c-45fc-bd06-c0f75a267960\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-nv6kl"
Oct 30 22:20:36.113310 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:36.113249    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/732b6214-aa8c-45fc-bd06-c0f75a267960-workload-socket\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-nv6kl\" (UID: \"732b6214-aa8c-45fc-bd06-c0f75a267960\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-nv6kl"
Oct 30 22:20:36.113310 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:36.113276    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9kmg\" (UniqueName: \"kubernetes.io/projected/732b6214-aa8c-45fc-bd06-c0f75a267960-kube-api-access-v9kmg\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-nv6kl\" (UID: \"732b6214-aa8c-45fc-bd06-c0f75a267960\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-nv6kl"
Oct 30 22:20:36.113420 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:36.113314    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/732b6214-aa8c-45fc-bd06-c0f75a267960-istio-data\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-nv6kl\" (UID: \"732b6214-aa8c-45fc-bd06-c0f75a267960\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-nv6kl"
Oct 30 22:20:36.113497 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:36.113449    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/732b6214-aa8c-45fc-bd06-c0f75a267960-istiod-ca-cert\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-nv6kl\" (UID: \"732b6214-aa8c-45fc-bd06-c0f75a267960\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-nv6kl"
Oct 30 22:20:36.113598 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:36.113563    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/732b6214-aa8c-45fc-bd06-c0f75a267960-workload-certs\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-nv6kl\" (UID: \"732b6214-aa8c-45fc-bd06-c0f75a267960\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-nv6kl"
Oct 30 22:20:36.113697 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:36.113676    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/732b6214-aa8c-45fc-bd06-c0f75a267960-workload-socket\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-nv6kl\" (UID: \"732b6214-aa8c-45fc-bd06-c0f75a267960\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-nv6kl"
Oct 30 22:20:36.115419 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:36.115393    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/732b6214-aa8c-45fc-bd06-c0f75a267960-istio-envoy\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-nv6kl\" (UID: \"732b6214-aa8c-45fc-bd06-c0f75a267960\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-nv6kl"
Oct 30 22:20:36.115733 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:36.115715    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/732b6214-aa8c-45fc-bd06-c0f75a267960-istio-podinfo\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-nv6kl\" (UID: \"732b6214-aa8c-45fc-bd06-c0f75a267960\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-nv6kl"
Oct 30 22:20:36.118841 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:36.118478    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-nv6kl"]
Oct 30 22:20:36.169008 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:36.168930    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/732b6214-aa8c-45fc-bd06-c0f75a267960-istio-token\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-nv6kl\" (UID: \"732b6214-aa8c-45fc-bd06-c0f75a267960\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-nv6kl"
Oct 30 22:20:36.169495 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:36.169473    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9kmg\" (UniqueName: \"kubernetes.io/projected/732b6214-aa8c-45fc-bd06-c0f75a267960-kube-api-access-v9kmg\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-nv6kl\" (UID: \"732b6214-aa8c-45fc-bd06-c0f75a267960\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-nv6kl"
Oct 30 22:20:36.231233 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:36.231189    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-nv6kl"
Oct 30 22:20:36.651374 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:36.651341    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-nv6kl"]
Oct 30 22:20:36.651701 ip-10-0-130-60 kubenswrapper[5061]: W1030 22:20:36.651675    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod732b6214_aa8c_45fc_bd06_c0f75a267960.slice/crio-f2550a67274b63f77056755c091d44a7944829a5f0bd0a35508871ab93a55ced WatchSource:0}: Error finding container f2550a67274b63f77056755c091d44a7944829a5f0bd0a35508871ab93a55ced: Status 404 returned error can't find the container with id f2550a67274b63f77056755c091d44a7944829a5f0bd0a35508871ab93a55ced
Oct 30 22:20:36.654570 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:36.654520    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:20:36.654668 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:36.654605    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:20:36.654668 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:36.654635    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:20:37.595960 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:37.595925    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-nv6kl" event={"ID":"732b6214-aa8c-45fc-bd06-c0f75a267960","Type":"ContainerStarted","Data":"9cb0b7e5f37d1b429283bcdb104977b0c36564f7cf0cc9f930c4493a4c93fe50"}
Oct 30 22:20:37.596360 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:37.595980    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-nv6kl" event={"ID":"732b6214-aa8c-45fc-bd06-c0f75a267960","Type":"ContainerStarted","Data":"f2550a67274b63f77056755c091d44a7944829a5f0bd0a35508871ab93a55ced"}
Oct 30 22:20:38.232761 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:38.232663    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-nv6kl"
Oct 30 22:20:38.234350 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:38.234316    5061 patch_prober.go:29] interesting pod/e2e-mt-778d39d3-openshift-default-bff74755c-nv6kl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body=
Oct 30 22:20:38.234474 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:38.234371    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-nv6kl" podUID="732b6214-aa8c-45fc-bd06-c0f75a267960" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused"
Oct 30 22:20:39.241664 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:39.241623    5061 patch_prober.go:29] interesting pod/e2e-mt-778d39d3-openshift-default-bff74755c-nv6kl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body=
Oct 30 22:20:39.242237 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:39.242203    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-nv6kl" podUID="732b6214-aa8c-45fc-bd06-c0f75a267960" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused"
Oct 30 22:20:40.088374 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:40.088318    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-nv6kl" podStartSLOduration=5.088303973 podStartE2EDuration="5.088303973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:20:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:20:37.732956001 +0000 UTC m=+2291.345725785" watchObservedRunningTime="2026-10-30 22:20:40.088303973 +0000 UTC m=+2293.701073730"
Oct 30 22:20:40.089894 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:40.089867    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0ce563-vqwjg"]
Oct 30 22:20:40.094603 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:40.094583    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0ce563-vqwjg"
Oct 30 22:20:40.143667 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:40.143636    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:20:40.173275 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:40.173234    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dccns\" (UniqueName: \"kubernetes.io/projected/6954b8cd-7365-435f-b281-d5339ac3df13-kube-api-access-dccns\") pod \"maas-api-revoke-keys-e2e-api-b-0ce563-vqwjg\" (UID: \"6954b8cd-7365-435f-b281-d5339ac3df13\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0ce563-vqwjg"
Oct 30 22:20:40.173454 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:40.173410    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6954b8cd-7365-435f-b281-d5339ac3df13-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-0ce563-vqwjg\" (UID: \"6954b8cd-7365-435f-b281-d5339ac3df13\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0ce563-vqwjg"
Oct 30 22:20:40.173548 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:40.173501    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6954b8cd-7365-435f-b281-d5339ac3df13-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-0ce563-vqwjg\" (UID: \"6954b8cd-7365-435f-b281-d5339ac3df13\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0ce563-vqwjg"
Oct 30 22:20:40.190646 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:40.190617    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gdmfg\""
Oct 30 22:20:40.194981 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:40.194957    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0ce563-vqwjg"]
Oct 30 22:20:40.232900 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:40.232870    5061 patch_prober.go:29] interesting pod/e2e-mt-778d39d3-openshift-default-bff74755c-nv6kl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body=
Oct 30 22:20:40.233061 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:40.232929    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-nv6kl" podUID="732b6214-aa8c-45fc-bd06-c0f75a267960" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused"
Oct 30 22:20:40.274786 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:40.274753    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6954b8cd-7365-435f-b281-d5339ac3df13-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-0ce563-vqwjg\" (UID: \"6954b8cd-7365-435f-b281-d5339ac3df13\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0ce563-vqwjg"
Oct 30 22:20:40.275140 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:40.274828    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6954b8cd-7365-435f-b281-d5339ac3df13-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-0ce563-vqwjg\" (UID: \"6954b8cd-7365-435f-b281-d5339ac3df13\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0ce563-vqwjg"
Oct 30 22:20:40.275140 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:40.274955    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dccns\" (UniqueName: \"kubernetes.io/projected/6954b8cd-7365-435f-b281-d5339ac3df13-kube-api-access-dccns\") pod \"maas-api-revoke-keys-e2e-api-b-0ce563-vqwjg\" (UID: \"6954b8cd-7365-435f-b281-d5339ac3df13\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0ce563-vqwjg"
Oct 30 22:20:40.275406 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:40.275385    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6954b8cd-7365-435f-b281-d5339ac3df13-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-0ce563-vqwjg\" (UID: \"6954b8cd-7365-435f-b281-d5339ac3df13\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0ce563-vqwjg"
Oct 30 22:20:40.277312 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:40.277278    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6954b8cd-7365-435f-b281-d5339ac3df13-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-0ce563-vqwjg\" (UID: \"6954b8cd-7365-435f-b281-d5339ac3df13\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0ce563-vqwjg"
Oct 30 22:20:40.333668 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:40.333628    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dccns\" (UniqueName: \"kubernetes.io/projected/6954b8cd-7365-435f-b281-d5339ac3df13-kube-api-access-dccns\") pod \"maas-api-revoke-keys-e2e-api-b-0ce563-vqwjg\" (UID: \"6954b8cd-7365-435f-b281-d5339ac3df13\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0ce563-vqwjg"
Oct 30 22:20:40.426094 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:40.425759    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0ce563-vqwjg"
Oct 30 22:20:40.800641 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:40.800615    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0ce563-vqwjg"]
Oct 30 22:20:40.802088 ip-10-0-130-60 kubenswrapper[5061]: W1030 22:20:40.802057    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6954b8cd_7365_435f_b281_d5339ac3df13.slice/crio-a4c76499d2648745a3a9f26635d18826b188722196020e7a8af3094cae16b071 WatchSource:0}: Error finding container a4c76499d2648745a3a9f26635d18826b188722196020e7a8af3094cae16b071: Status 404 returned error can't find the container with id a4c76499d2648745a3a9f26635d18826b188722196020e7a8af3094cae16b071
Oct 30 22:20:41.233847 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:41.233813    5061 patch_prober.go:29] interesting pod/e2e-mt-778d39d3-openshift-default-bff74755c-nv6kl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body=
Oct 30 22:20:41.234032 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:41.233874    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-nv6kl" podUID="732b6214-aa8c-45fc-bd06-c0f75a267960" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused"
Oct 30 22:20:41.649775 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:41.649734    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0ce563-vqwjg" event={"ID":"6954b8cd-7365-435f-b281-d5339ac3df13","Type":"ContainerStarted","Data":"5edae1f59bc47fb50fef948d15ed8baff418955014b67f38a74cc1f224b539de"}
Oct 30 22:20:41.650148 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:41.649804    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0ce563-vqwjg" event={"ID":"6954b8cd-7365-435f-b281-d5339ac3df13","Type":"ContainerStarted","Data":"a4c76499d2648745a3a9f26635d18826b188722196020e7a8af3094cae16b071"}
Oct 30 22:20:41.770222 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:41.770164    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0ce563-vqwjg" podStartSLOduration=1.770151389 podStartE2EDuration="1.770151389s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:20:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:20:41.766665399 +0000 UTC m=+2295.379435169" watchObservedRunningTime="2026-10-30 22:20:41.770151389 +0000 UTC m=+2295.382921147"
Oct 30 22:20:42.234648 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:42.232453    5061 patch_prober.go:29] interesting pod/e2e-mt-778d39d3-openshift-default-bff74755c-nv6kl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body=
Oct 30 22:20:42.234648 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:42.232511    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-nv6kl" podUID="732b6214-aa8c-45fc-bd06-c0f75a267960" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused"
Oct 30 22:20:42.661572 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:42.661511    5061 generic.go:356] "Generic (PLEG): container finished" podID="6954b8cd-7365-435f-b281-d5339ac3df13" containerID="5edae1f59bc47fb50fef948d15ed8baff418955014b67f38a74cc1f224b539de" exitCode=0
Oct 30 22:20:42.662032 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:42.661635    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0ce563-vqwjg" event={"ID":"6954b8cd-7365-435f-b281-d5339ac3df13","Type":"ContainerDied","Data":"5edae1f59bc47fb50fef948d15ed8baff418955014b67f38a74cc1f224b539de"}
Oct 30 22:20:43.236123 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:43.236087    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-nv6kl"
Oct 30 22:20:43.236582 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:43.236559    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-nv6kl"
Oct 30 22:20:43.237303 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:43.237285    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-nv6kl"
Oct 30 22:20:43.836041 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:43.836017    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0ce563-vqwjg"
Oct 30 22:20:43.929230 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:43.929147    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6954b8cd-7365-435f-b281-d5339ac3df13-kube-api-access-dccns\" (UniqueName: \"kubernetes.io/projected/6954b8cd-7365-435f-b281-d5339ac3df13-kube-api-access-dccns\") pod \"6954b8cd-7365-435f-b281-d5339ac3df13\" (UID: \"6954b8cd-7365-435f-b281-d5339ac3df13\") "
Oct 30 22:20:43.929230 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:43.929217    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6954b8cd-7365-435f-b281-d5339ac3df13-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6954b8cd-7365-435f-b281-d5339ac3df13-maas-api-openshift-ca\") pod \"6954b8cd-7365-435f-b281-d5339ac3df13\" (UID: \"6954b8cd-7365-435f-b281-d5339ac3df13\") "
Oct 30 22:20:43.929421 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:43.929366    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6954b8cd-7365-435f-b281-d5339ac3df13-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6954b8cd-7365-435f-b281-d5339ac3df13-maas-api-xks-ca\") pod \"6954b8cd-7365-435f-b281-d5339ac3df13\" (UID: \"6954b8cd-7365-435f-b281-d5339ac3df13\") "
Oct 30 22:20:43.929639 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:43.929617    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6954b8cd-7365-435f-b281-d5339ac3df13-maas-api-openshift-ca" pod "6954b8cd-7365-435f-b281-d5339ac3df13" (UID: "6954b8cd-7365-435f-b281-d5339ac3df13"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:20:43.929936 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:43.929914    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6954b8cd-7365-435f-b281-d5339ac3df13-maas-api-openshift-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:20:43.931472 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:43.931450    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6954b8cd-7365-435f-b281-d5339ac3df13-maas-api-xks-ca" pod "6954b8cd-7365-435f-b281-d5339ac3df13" (UID: "6954b8cd-7365-435f-b281-d5339ac3df13"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:20:43.931472 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:43.931454    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6954b8cd-7365-435f-b281-d5339ac3df13-kube-api-access-dccns" pod "6954b8cd-7365-435f-b281-d5339ac3df13" (UID: "6954b8cd-7365-435f-b281-d5339ac3df13"). InnerVolumeSpecName "kube-api-access-dccns". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:20:44.030929 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:44.030900    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dccns\" (UniqueName: \"kubernetes.io/projected/6954b8cd-7365-435f-b281-d5339ac3df13-kube-api-access-dccns\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:20:44.030929 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:44.030927    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6954b8cd-7365-435f-b281-d5339ac3df13-maas-api-xks-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:20:44.677631 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:44.677602    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0ce563-vqwjg"
Oct 30 22:20:44.677631 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:44.677618    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0ce563-vqwjg" event={"ID":"6954b8cd-7365-435f-b281-d5339ac3df13","Type":"ContainerDied","Data":"a4c76499d2648745a3a9f26635d18826b188722196020e7a8af3094cae16b071"}
Oct 30 22:20:44.677833 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:44.677668    5061 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a4c76499d2648745a3a9f26635d18826b188722196020e7a8af3094cae16b071"
Oct 30 22:20:45.804102 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:45.804070    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-778d39d3-4txvg"]
Oct 30 22:20:45.804934 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:45.804911    5061 cpu_manager.go:414] "containerMap: removing container" podUID="6954b8cd-7365-435f-b281-d5339ac3df13" containerName="revoke-keys"
Oct 30 22:20:45.804934 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:45.804933    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6954b8cd-7365-435f-b281-d5339ac3df13" containerName="revoke-keys"
Oct 30 22:20:45.805098 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:45.805063    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="6954b8cd-7365-435f-b281-d5339ac3df13" containerName="revoke-keys"
Oct 30 22:20:45.808873 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:45.808846    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-778d39d3-4txvg"
Oct 30 22:20:45.853113 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:45.853070    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/339da026-9f6c-48d0-8d77-6f2fd7675c75-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-778d39d3-4txvg\" (UID: \"339da026-9f6c-48d0-8d77-6f2fd7675c75\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-778d39d3-4txvg"
Oct 30 22:20:45.853276 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:45.853207    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rsv9\" (UniqueName: \"kubernetes.io/projected/339da026-9f6c-48d0-8d77-6f2fd7675c75-kube-api-access-7rsv9\") pod \"maas-api-revoke-keys-e2e-mt-778d39d3-4txvg\" (UID: \"339da026-9f6c-48d0-8d77-6f2fd7675c75\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-778d39d3-4txvg"
Oct 30 22:20:45.853344 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:45.853297    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/339da026-9f6c-48d0-8d77-6f2fd7675c75-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-778d39d3-4txvg\" (UID: \"339da026-9f6c-48d0-8d77-6f2fd7675c75\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-778d39d3-4txvg"
Oct 30 22:20:45.908144 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:45.908112    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:20:45.908337 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:45.908318    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gdmfg\""
Oct 30 22:20:45.912262 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:45.912238    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-778d39d3-4txvg"]
Oct 30 22:20:45.955365 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:45.955331    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rsv9\" (UniqueName: \"kubernetes.io/projected/339da026-9f6c-48d0-8d77-6f2fd7675c75-kube-api-access-7rsv9\") pod \"maas-api-revoke-keys-e2e-mt-778d39d3-4txvg\" (UID: \"339da026-9f6c-48d0-8d77-6f2fd7675c75\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-778d39d3-4txvg"
Oct 30 22:20:45.955551 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:45.955427    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/339da026-9f6c-48d0-8d77-6f2fd7675c75-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-778d39d3-4txvg\" (UID: \"339da026-9f6c-48d0-8d77-6f2fd7675c75\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-778d39d3-4txvg"
Oct 30 22:20:45.955610 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:45.955556    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/339da026-9f6c-48d0-8d77-6f2fd7675c75-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-778d39d3-4txvg\" (UID: \"339da026-9f6c-48d0-8d77-6f2fd7675c75\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-778d39d3-4txvg"
Oct 30 22:20:45.956228 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:45.956207    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/339da026-9f6c-48d0-8d77-6f2fd7675c75-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-778d39d3-4txvg\" (UID: \"339da026-9f6c-48d0-8d77-6f2fd7675c75\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-778d39d3-4txvg"
Oct 30 22:20:45.958304 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:45.958278    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/339da026-9f6c-48d0-8d77-6f2fd7675c75-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-778d39d3-4txvg\" (UID: \"339da026-9f6c-48d0-8d77-6f2fd7675c75\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-778d39d3-4txvg"
Oct 30 22:20:46.011726 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:46.011690    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rsv9\" (UniqueName: \"kubernetes.io/projected/339da026-9f6c-48d0-8d77-6f2fd7675c75-kube-api-access-7rsv9\") pod \"maas-api-revoke-keys-e2e-mt-778d39d3-4txvg\" (UID: \"339da026-9f6c-48d0-8d77-6f2fd7675c75\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-778d39d3-4txvg"
Oct 30 22:20:46.123498 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:46.123409    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-778d39d3-4txvg"
Oct 30 22:20:46.504885 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:46.504850    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-778d39d3-4txvg"]
Oct 30 22:20:46.506473 ip-10-0-130-60 kubenswrapper[5061]: W1030 22:20:46.506434    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod339da026_9f6c_48d0_8d77_6f2fd7675c75.slice/crio-2dd2f4513c97c4dcc6e1aa69b4387623963261909134cb0f8e03af6af7acca5b WatchSource:0}: Error finding container 2dd2f4513c97c4dcc6e1aa69b4387623963261909134cb0f8e03af6af7acca5b: Status 404 returned error can't find the container with id 2dd2f4513c97c4dcc6e1aa69b4387623963261909134cb0f8e03af6af7acca5b
Oct 30 22:20:46.694717 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:46.694680    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-778d39d3-4txvg" event={"ID":"339da026-9f6c-48d0-8d77-6f2fd7675c75","Type":"ContainerStarted","Data":"2dd2f4513c97c4dcc6e1aa69b4387623963261909134cb0f8e03af6af7acca5b"}
Oct 30 22:20:47.703554 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:47.703429    5061 generic.go:356] "Generic (PLEG): container finished" podID="339da026-9f6c-48d0-8d77-6f2fd7675c75" containerID="0acb61a5a99201b677ce533536f7ca512d81052d1ff57fce086ea4eca0d1f852" exitCode=0
Oct 30 22:20:47.703935 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:47.703559    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-778d39d3-4txvg" event={"ID":"339da026-9f6c-48d0-8d77-6f2fd7675c75","Type":"ContainerDied","Data":"0acb61a5a99201b677ce533536f7ca512d81052d1ff57fce086ea4eca0d1f852"}
Oct 30 22:20:48.864723 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:48.864699    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-778d39d3-4txvg"
Oct 30 22:20:48.995762 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:48.995687    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/339da026-9f6c-48d0-8d77-6f2fd7675c75-kube-api-access-7rsv9\" (UniqueName: \"kubernetes.io/projected/339da026-9f6c-48d0-8d77-6f2fd7675c75-kube-api-access-7rsv9\") pod \"339da026-9f6c-48d0-8d77-6f2fd7675c75\" (UID: \"339da026-9f6c-48d0-8d77-6f2fd7675c75\") "
Oct 30 22:20:48.995927 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:48.995804    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/339da026-9f6c-48d0-8d77-6f2fd7675c75-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/339da026-9f6c-48d0-8d77-6f2fd7675c75-maas-api-xks-ca\") pod \"339da026-9f6c-48d0-8d77-6f2fd7675c75\" (UID: \"339da026-9f6c-48d0-8d77-6f2fd7675c75\") "
Oct 30 22:20:48.996001 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:48.995983    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/339da026-9f6c-48d0-8d77-6f2fd7675c75-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/339da026-9f6c-48d0-8d77-6f2fd7675c75-maas-api-openshift-ca\") pod \"339da026-9f6c-48d0-8d77-6f2fd7675c75\" (UID: \"339da026-9f6c-48d0-8d77-6f2fd7675c75\") "
Oct 30 22:20:48.996309 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:48.996283    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/339da026-9f6c-48d0-8d77-6f2fd7675c75-maas-api-openshift-ca" pod "339da026-9f6c-48d0-8d77-6f2fd7675c75" (UID: "339da026-9f6c-48d0-8d77-6f2fd7675c75"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:20:48.996540 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:48.996510    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/339da026-9f6c-48d0-8d77-6f2fd7675c75-maas-api-openshift-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:20:48.997985 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:48.997959    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/339da026-9f6c-48d0-8d77-6f2fd7675c75-kube-api-access-7rsv9" pod "339da026-9f6c-48d0-8d77-6f2fd7675c75" (UID: "339da026-9f6c-48d0-8d77-6f2fd7675c75"). InnerVolumeSpecName "kube-api-access-7rsv9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:20:48.998773 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:48.998749    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/339da026-9f6c-48d0-8d77-6f2fd7675c75-maas-api-xks-ca" pod "339da026-9f6c-48d0-8d77-6f2fd7675c75" (UID: "339da026-9f6c-48d0-8d77-6f2fd7675c75"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:20:49.097866 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:49.097832    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rsv9\" (UniqueName: \"kubernetes.io/projected/339da026-9f6c-48d0-8d77-6f2fd7675c75-kube-api-access-7rsv9\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:20:49.097866 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:49.097860    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/339da026-9f6c-48d0-8d77-6f2fd7675c75-maas-api-xks-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:20:49.717986 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:49.717956    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-778d39d3-4txvg"
Oct 30 22:20:49.718190 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:49.717984    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-778d39d3-4txvg" event={"ID":"339da026-9f6c-48d0-8d77-6f2fd7675c75","Type":"ContainerDied","Data":"2dd2f4513c97c4dcc6e1aa69b4387623963261909134cb0f8e03af6af7acca5b"}
Oct 30 22:20:49.718190 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:49.718033    5061 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2dd2f4513c97c4dcc6e1aa69b4387623963261909134cb0f8e03af6af7acca5b"
Oct 30 22:20:50.450625 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:50.450570    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bef575f8-5sfmq"]
Oct 30 22:20:50.451662 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:50.451636    5061 cpu_manager.go:414] "containerMap: removing container" podUID="339da026-9f6c-48d0-8d77-6f2fd7675c75" containerName="revoke-keys"
Oct 30 22:20:50.451662 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:50.451660    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="339da026-9f6c-48d0-8d77-6f2fd7675c75" containerName="revoke-keys"
Oct 30 22:20:50.451888 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:50.451875    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="339da026-9f6c-48d0-8d77-6f2fd7675c75" containerName="revoke-keys"
Oct 30 22:20:50.455601 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:50.455578    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bef575f8-5sfmq"
Oct 30 22:20:50.505424 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:50.505395    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:20:50.513242 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:50.513211    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4402ea44-8969-45e8-8037-593633026e9e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bef575f8-5sfmq\" (UID: \"4402ea44-8969-45e8-8037-593633026e9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bef575f8-5sfmq"
Oct 30 22:20:50.513387 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:50.513337    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vc25\" (UniqueName: \"kubernetes.io/projected/4402ea44-8969-45e8-8037-593633026e9e-kube-api-access-7vc25\") pod \"maas-api-revoke-sub-bef575f8-5sfmq\" (UID: \"4402ea44-8969-45e8-8037-593633026e9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bef575f8-5sfmq"
Oct 30 22:20:50.513654 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:50.513624    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4402ea44-8969-45e8-8037-593633026e9e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bef575f8-5sfmq\" (UID: \"4402ea44-8969-45e8-8037-593633026e9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bef575f8-5sfmq"
Oct 30 22:20:50.552146 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:50.552108    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gdmfg\""
Oct 30 22:20:50.556645 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:50.556614    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bef575f8-5sfmq"]
Oct 30 22:20:50.615282 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:50.615246    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vc25\" (UniqueName: \"kubernetes.io/projected/4402ea44-8969-45e8-8037-593633026e9e-kube-api-access-7vc25\") pod \"maas-api-revoke-sub-bef575f8-5sfmq\" (UID: \"4402ea44-8969-45e8-8037-593633026e9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bef575f8-5sfmq"
Oct 30 22:20:50.615678 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:50.615651    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4402ea44-8969-45e8-8037-593633026e9e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bef575f8-5sfmq\" (UID: \"4402ea44-8969-45e8-8037-593633026e9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bef575f8-5sfmq"
Oct 30 22:20:50.615864 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:50.615844    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4402ea44-8969-45e8-8037-593633026e9e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bef575f8-5sfmq\" (UID: \"4402ea44-8969-45e8-8037-593633026e9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bef575f8-5sfmq"
Oct 30 22:20:50.616370 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:50.616346    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4402ea44-8969-45e8-8037-593633026e9e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bef575f8-5sfmq\" (UID: \"4402ea44-8969-45e8-8037-593633026e9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bef575f8-5sfmq"
Oct 30 22:20:50.618380 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:50.618357    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4402ea44-8969-45e8-8037-593633026e9e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bef575f8-5sfmq\" (UID: \"4402ea44-8969-45e8-8037-593633026e9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bef575f8-5sfmq"
Oct 30 22:20:50.669643 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:50.669611    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vc25\" (UniqueName: \"kubernetes.io/projected/4402ea44-8969-45e8-8037-593633026e9e-kube-api-access-7vc25\") pod \"maas-api-revoke-sub-bef575f8-5sfmq\" (UID: \"4402ea44-8969-45e8-8037-593633026e9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bef575f8-5sfmq"
Oct 30 22:20:50.772989 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:50.772922    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bef575f8-5sfmq"
Oct 30 22:20:51.059483 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:51.059449    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-k5knf"]
Oct 30 22:20:51.064920 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:51.064899    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-k5knf"
Oct 30 22:20:51.122674 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:51.122628    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3489262b-293e-4dba-8bb2-efa4333ab66d-workload-socket\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-k5knf\" (UID: \"3489262b-293e-4dba-8bb2-efa4333ab66d\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-k5knf"
Oct 30 22:20:51.122829 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:51.122773    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3489262b-293e-4dba-8bb2-efa4333ab66d-workload-certs\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-k5knf\" (UID: \"3489262b-293e-4dba-8bb2-efa4333ab66d\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-k5knf"
Oct 30 22:20:51.122929 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:51.122904    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3489262b-293e-4dba-8bb2-efa4333ab66d-istio-token\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-k5knf\" (UID: \"3489262b-293e-4dba-8bb2-efa4333ab66d\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-k5knf"
Oct 30 22:20:51.123122 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:51.123019    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3489262b-293e-4dba-8bb2-efa4333ab66d-istio-envoy\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-k5knf\" (UID: \"3489262b-293e-4dba-8bb2-efa4333ab66d\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-k5knf"
Oct 30 22:20:51.123122 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:51.123059    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3489262b-293e-4dba-8bb2-efa4333ab66d-credential-socket\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-k5knf\" (UID: \"3489262b-293e-4dba-8bb2-efa4333ab66d\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-k5knf"
Oct 30 22:20:51.123122 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:51.123092    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3489262b-293e-4dba-8bb2-efa4333ab66d-istio-podinfo\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-k5knf\" (UID: \"3489262b-293e-4dba-8bb2-efa4333ab66d\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-k5knf"
Oct 30 22:20:51.123371 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:51.123142    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3489262b-293e-4dba-8bb2-efa4333ab66d-istio-data\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-k5knf\" (UID: \"3489262b-293e-4dba-8bb2-efa4333ab66d\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-k5knf"
Oct 30 22:20:51.123371 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:51.123200    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3489262b-293e-4dba-8bb2-efa4333ab66d-istiod-ca-cert\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-k5knf\" (UID: \"3489262b-293e-4dba-8bb2-efa4333ab66d\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-k5knf"
Oct 30 22:20:51.123371 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:51.123245    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dg4j\" (UniqueName: \"kubernetes.io/projected/3489262b-293e-4dba-8bb2-efa4333ab66d-kube-api-access-9dg4j\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-k5knf\" (UID: \"3489262b-293e-4dba-8bb2-efa4333ab66d\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-k5knf"
Oct 30 22:20:51.168437 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:51.168410    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bef575f8-5sfmq"]
Oct 30 22:20:51.189652 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:51.189623    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-k5knf"]
Oct 30 22:20:51.224978 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:51.224947    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3489262b-293e-4dba-8bb2-efa4333ab66d-istio-envoy\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-k5knf\" (UID: \"3489262b-293e-4dba-8bb2-efa4333ab66d\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-k5knf"
Oct 30 22:20:51.225130 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:51.224996    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3489262b-293e-4dba-8bb2-efa4333ab66d-credential-socket\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-k5knf\" (UID: \"3489262b-293e-4dba-8bb2-efa4333ab66d\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-k5knf"
Oct 30 22:20:51.225130 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:51.225031    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3489262b-293e-4dba-8bb2-efa4333ab66d-istio-podinfo\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-k5knf\" (UID: \"3489262b-293e-4dba-8bb2-efa4333ab66d\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-k5knf"
Oct 30 22:20:51.225130 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:51.225074    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3489262b-293e-4dba-8bb2-efa4333ab66d-istio-data\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-k5knf\" (UID: \"3489262b-293e-4dba-8bb2-efa4333ab66d\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-k5knf"
Oct 30 22:20:51.225270 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:51.225132    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3489262b-293e-4dba-8bb2-efa4333ab66d-istiod-ca-cert\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-k5knf\" (UID: \"3489262b-293e-4dba-8bb2-efa4333ab66d\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-k5knf"
Oct 30 22:20:51.225270 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:51.225175    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dg4j\" (UniqueName: \"kubernetes.io/projected/3489262b-293e-4dba-8bb2-efa4333ab66d-kube-api-access-9dg4j\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-k5knf\" (UID: \"3489262b-293e-4dba-8bb2-efa4333ab66d\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-k5knf"
Oct 30 22:20:51.225383 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:51.225291    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3489262b-293e-4dba-8bb2-efa4333ab66d-workload-socket\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-k5knf\" (UID: \"3489262b-293e-4dba-8bb2-efa4333ab66d\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-k5knf"
Oct 30 22:20:51.225383 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:51.225347    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3489262b-293e-4dba-8bb2-efa4333ab66d-workload-certs\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-k5knf\" (UID: \"3489262b-293e-4dba-8bb2-efa4333ab66d\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-k5knf"
Oct 30 22:20:51.225499 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:51.225412    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3489262b-293e-4dba-8bb2-efa4333ab66d-istio-token\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-k5knf\" (UID: \"3489262b-293e-4dba-8bb2-efa4333ab66d\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-k5knf"
Oct 30 22:20:51.225579 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:51.225508    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3489262b-293e-4dba-8bb2-efa4333ab66d-istio-data\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-k5knf\" (UID: \"3489262b-293e-4dba-8bb2-efa4333ab66d\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-k5knf"
Oct 30 22:20:51.225822 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:51.225794    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3489262b-293e-4dba-8bb2-efa4333ab66d-workload-socket\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-k5knf\" (UID: \"3489262b-293e-4dba-8bb2-efa4333ab66d\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-k5knf"
Oct 30 22:20:51.225913 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:51.225881    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3489262b-293e-4dba-8bb2-efa4333ab66d-workload-certs\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-k5knf\" (UID: \"3489262b-293e-4dba-8bb2-efa4333ab66d\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-k5knf"
Oct 30 22:20:51.225980 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:51.225924    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3489262b-293e-4dba-8bb2-efa4333ab66d-istiod-ca-cert\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-k5knf\" (UID: \"3489262b-293e-4dba-8bb2-efa4333ab66d\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-k5knf"
Oct 30 22:20:51.225980 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:51.225968    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3489262b-293e-4dba-8bb2-efa4333ab66d-credential-socket\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-k5knf\" (UID: \"3489262b-293e-4dba-8bb2-efa4333ab66d\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-k5knf"
Oct 30 22:20:51.227590 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:51.227568    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3489262b-293e-4dba-8bb2-efa4333ab66d-istio-envoy\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-k5knf\" (UID: \"3489262b-293e-4dba-8bb2-efa4333ab66d\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-k5knf"
Oct 30 22:20:51.227864 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:51.227846    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3489262b-293e-4dba-8bb2-efa4333ab66d-istio-podinfo\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-k5knf\" (UID: \"3489262b-293e-4dba-8bb2-efa4333ab66d\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-k5knf"
Oct 30 22:20:51.280820 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:51.280794    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dg4j\" (UniqueName: \"kubernetes.io/projected/3489262b-293e-4dba-8bb2-efa4333ab66d-kube-api-access-9dg4j\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-k5knf\" (UID: \"3489262b-293e-4dba-8bb2-efa4333ab66d\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-k5knf"
Oct 30 22:20:51.282320 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:51.282291    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3489262b-293e-4dba-8bb2-efa4333ab66d-istio-token\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-k5knf\" (UID: \"3489262b-293e-4dba-8bb2-efa4333ab66d\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-k5knf"
Oct 30 22:20:51.379680 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:51.379594    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-k5knf"
Oct 30 22:20:51.505389 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:51.505355    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0ce563-f9b586484-rg56t"]
Oct 30 22:20:51.602456 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:51.602419    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0ce563-6b8c6c95bc-fsrrg"]
Oct 30 22:20:51.734858 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:51.734756    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bef575f8-5sfmq" event={"ID":"4402ea44-8969-45e8-8037-593633026e9e","Type":"ContainerStarted","Data":"6a8a0277544b978e7ebb53d6dafedcdbf49237cd4c56520a3904176fa032b365"}
Oct 30 22:20:51.734858 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:51.734813    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bef575f8-5sfmq" event={"ID":"4402ea44-8969-45e8-8037-593633026e9e","Type":"ContainerStarted","Data":"81340a80326ea9a32a2d32f1bc815ce55fd36321ce0d472bee10f8049d97b333"}
Oct 30 22:20:51.810338 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:51.810293    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-k5knf"]
Oct 30 22:20:51.816585 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:51.816521    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:20:51.816692 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:51.816632    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:20:51.816692 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:51.816680    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:20:51.851617 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:51.851570    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bef575f8-5sfmq" podStartSLOduration=1.851552464 podStartE2EDuration="1.851552464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:20:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:20:51.848522298 +0000 UTC m=+2305.461292056" watchObservedRunningTime="2026-10-30 22:20:51.851552464 +0000 UTC m=+2305.464322222"
Oct 30 22:20:52.744610 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:52.744568    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-k5knf" event={"ID":"3489262b-293e-4dba-8bb2-efa4333ab66d","Type":"ContainerStarted","Data":"69650f186337a304c285b07f94defc757c4f41455ed19bfc01752e056d037b6c"}
Oct 30 22:20:52.745096 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:52.744640    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-k5knf" event={"ID":"3489262b-293e-4dba-8bb2-efa4333ab66d","Type":"ContainerStarted","Data":"6b537cdbf17f3ddd2abc593fd2fd5b51dff360bcf94560929176170c069d8cfa"}
Oct 30 22:20:52.746360 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:52.746323    5061 generic.go:356] "Generic (PLEG): container finished" podID="4402ea44-8969-45e8-8037-593633026e9e" containerID="6a8a0277544b978e7ebb53d6dafedcdbf49237cd4c56520a3904176fa032b365" exitCode=0
Oct 30 22:20:52.746500 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:52.746408    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bef575f8-5sfmq" event={"ID":"4402ea44-8969-45e8-8037-593633026e9e","Type":"ContainerDied","Data":"6a8a0277544b978e7ebb53d6dafedcdbf49237cd4c56520a3904176fa032b365"}
Oct 30 22:20:52.856585 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:52.855668    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c586c"]
Oct 30 22:20:52.856585 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:52.855915    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c586c" podUID="dc212ddd-b71e-44c3-88f2-f7c0936bf7d4" containerName="discovery" containerID="cri-o://4ae4b0264f6e207c137533a7efb2e4a0332771b10321bfa1e5bf7a7e9f57aba1" gracePeriod=30
Oct 30 22:20:52.872436 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:52.872391    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-k5knf" podStartSLOduration=1.8723759150000001 podStartE2EDuration="1.872375915s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:20:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:20:52.870384768 +0000 UTC m=+2306.483154522" watchObservedRunningTime="2026-10-30 22:20:52.872375915 +0000 UTC m=+2306.485145672"
Oct 30 22:20:53.380621 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:53.380416    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-k5knf"
Oct 30 22:20:53.381841 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:53.381808    5061 patch_prober.go:29] interesting pod/e2e-mt-778d39d3-openshift-default-bff74755c-k5knf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body=
Oct 30 22:20:53.381982 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:53.381858    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-k5knf" podUID="3489262b-293e-4dba-8bb2-efa4333ab66d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused"
Oct 30 22:20:53.450442 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:53.450416    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c586c"
Oct 30 22:20:53.596540 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:53.596493    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc212ddd-b71e-44c3-88f2-f7c0936bf7d4-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/dc212ddd-b71e-44c3-88f2-f7c0936bf7d4-istio-csr-dns-cert\") pod \"dc212ddd-b71e-44c3-88f2-f7c0936bf7d4\" (UID: \"dc212ddd-b71e-44c3-88f2-f7c0936bf7d4\") "
Oct 30 22:20:53.596710 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:53.596612    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc212ddd-b71e-44c3-88f2-f7c0936bf7d4-cacerts\" (UniqueName: \"kubernetes.io/secret/dc212ddd-b71e-44c3-88f2-f7c0936bf7d4-cacerts\") pod \"dc212ddd-b71e-44c3-88f2-f7c0936bf7d4\" (UID: \"dc212ddd-b71e-44c3-88f2-f7c0936bf7d4\") "
Oct 30 22:20:53.596797 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:53.596727    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc212ddd-b71e-44c3-88f2-f7c0936bf7d4-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/dc212ddd-b71e-44c3-88f2-f7c0936bf7d4-istio-csr-ca-configmap\") pod \"dc212ddd-b71e-44c3-88f2-f7c0936bf7d4\" (UID: \"dc212ddd-b71e-44c3-88f2-f7c0936bf7d4\") "
Oct 30 22:20:53.596797 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:53.596762    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc212ddd-b71e-44c3-88f2-f7c0936bf7d4-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc212ddd-b71e-44c3-88f2-f7c0936bf7d4-local-certs\") pod \"dc212ddd-b71e-44c3-88f2-f7c0936bf7d4\" (UID: \"dc212ddd-b71e-44c3-88f2-f7c0936bf7d4\") "
Oct 30 22:20:53.596917 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:53.596808    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc212ddd-b71e-44c3-88f2-f7c0936bf7d4-kube-api-access-86tlw\" (UniqueName: \"kubernetes.io/projected/dc212ddd-b71e-44c3-88f2-f7c0936bf7d4-kube-api-access-86tlw\") pod \"dc212ddd-b71e-44c3-88f2-f7c0936bf7d4\" (UID: \"dc212ddd-b71e-44c3-88f2-f7c0936bf7d4\") "
Oct 30 22:20:53.596917 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:53.596871    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc212ddd-b71e-44c3-88f2-f7c0936bf7d4-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/dc212ddd-b71e-44c3-88f2-f7c0936bf7d4-istio-kubeconfig\") pod \"dc212ddd-b71e-44c3-88f2-f7c0936bf7d4\" (UID: \"dc212ddd-b71e-44c3-88f2-f7c0936bf7d4\") "
Oct 30 22:20:53.597021 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:53.596974    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc212ddd-b71e-44c3-88f2-f7c0936bf7d4-istio-token\" (UniqueName: \"kubernetes.io/projected/dc212ddd-b71e-44c3-88f2-f7c0936bf7d4-istio-token\") pod \"dc212ddd-b71e-44c3-88f2-f7c0936bf7d4\" (UID: \"dc212ddd-b71e-44c3-88f2-f7c0936bf7d4\") "
Oct 30 22:20:53.597904 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:53.597870    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc212ddd-b71e-44c3-88f2-f7c0936bf7d4-istio-csr-ca-configmap" pod "dc212ddd-b71e-44c3-88f2-f7c0936bf7d4" (UID: "dc212ddd-b71e-44c3-88f2-f7c0936bf7d4"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:20:53.599470 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:53.599444    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc212ddd-b71e-44c3-88f2-f7c0936bf7d4-cacerts" pod "dc212ddd-b71e-44c3-88f2-f7c0936bf7d4" (UID: "dc212ddd-b71e-44c3-88f2-f7c0936bf7d4"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:20:53.599835 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:53.599811    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc212ddd-b71e-44c3-88f2-f7c0936bf7d4-local-certs" pod "dc212ddd-b71e-44c3-88f2-f7c0936bf7d4" (UID: "dc212ddd-b71e-44c3-88f2-f7c0936bf7d4"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:20:53.600320 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:53.600289    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc212ddd-b71e-44c3-88f2-f7c0936bf7d4-istio-token" pod "dc212ddd-b71e-44c3-88f2-f7c0936bf7d4" (UID: "dc212ddd-b71e-44c3-88f2-f7c0936bf7d4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:20:53.600632 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:53.600605    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc212ddd-b71e-44c3-88f2-f7c0936bf7d4-istio-csr-dns-cert" pod "dc212ddd-b71e-44c3-88f2-f7c0936bf7d4" (UID: "dc212ddd-b71e-44c3-88f2-f7c0936bf7d4"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:20:53.601574 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:53.601519    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc212ddd-b71e-44c3-88f2-f7c0936bf7d4-istio-kubeconfig" pod "dc212ddd-b71e-44c3-88f2-f7c0936bf7d4" (UID: "dc212ddd-b71e-44c3-88f2-f7c0936bf7d4"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:20:53.601574 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:53.601557    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc212ddd-b71e-44c3-88f2-f7c0936bf7d4-kube-api-access-86tlw" pod "dc212ddd-b71e-44c3-88f2-f7c0936bf7d4" (UID: "dc212ddd-b71e-44c3-88f2-f7c0936bf7d4"). InnerVolumeSpecName "kube-api-access-86tlw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:20:53.703931 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:53.703890    5061 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/dc212ddd-b71e-44c3-88f2-f7c0936bf7d4-istio-kubeconfig\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:20:53.703931 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:53.703923    5061 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc212ddd-b71e-44c3-88f2-f7c0936bf7d4-istio-token\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:20:53.704139 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:53.703938    5061 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/dc212ddd-b71e-44c3-88f2-f7c0936bf7d4-istio-csr-dns-cert\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:20:53.704139 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:53.703951    5061 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/dc212ddd-b71e-44c3-88f2-f7c0936bf7d4-cacerts\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:20:53.704139 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:53.703968    5061 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/dc212ddd-b71e-44c3-88f2-f7c0936bf7d4-istio-csr-ca-configmap\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:20:53.704139 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:53.703983    5061 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc212ddd-b71e-44c3-88f2-f7c0936bf7d4-local-certs\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:20:53.704139 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:53.703997    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86tlw\" (UniqueName: \"kubernetes.io/projected/dc212ddd-b71e-44c3-88f2-f7c0936bf7d4-kube-api-access-86tlw\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:20:53.755173 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:53.755140    5061 generic.go:356] "Generic (PLEG): container finished" podID="dc212ddd-b71e-44c3-88f2-f7c0936bf7d4" containerID="4ae4b0264f6e207c137533a7efb2e4a0332771b10321bfa1e5bf7a7e9f57aba1" exitCode=0
Oct 30 22:20:53.755573 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:53.755197    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c586c"
Oct 30 22:20:53.755573 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:53.755255    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c586c" event={"ID":"dc212ddd-b71e-44c3-88f2-f7c0936bf7d4","Type":"ContainerDied","Data":"4ae4b0264f6e207c137533a7efb2e4a0332771b10321bfa1e5bf7a7e9f57aba1"}
Oct 30 22:20:53.755573 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:53.755332    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c586c" event={"ID":"dc212ddd-b71e-44c3-88f2-f7c0936bf7d4","Type":"ContainerDied","Data":"1480db6712e38db03efeb5a5a8a01146be165ef5e7519849301c790993b59636"}
Oct 30 22:20:53.755573 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:53.755357    5061 scope.go:122] "RemoveContainer" containerID="4ae4b0264f6e207c137533a7efb2e4a0332771b10321bfa1e5bf7a7e9f57aba1"
Oct 30 22:20:53.768106 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:53.768031    5061 scope.go:122] "RemoveContainer" containerID="4ae4b0264f6e207c137533a7efb2e4a0332771b10321bfa1e5bf7a7e9f57aba1"
Oct 30 22:20:53.768367 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:20:53.768347    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ae4b0264f6e207c137533a7efb2e4a0332771b10321bfa1e5bf7a7e9f57aba1\": container with ID starting with 4ae4b0264f6e207c137533a7efb2e4a0332771b10321bfa1e5bf7a7e9f57aba1 not found: ID does not exist" containerID="4ae4b0264f6e207c137533a7efb2e4a0332771b10321bfa1e5bf7a7e9f57aba1"
Oct 30 22:20:53.768433 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:53.768378    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ae4b0264f6e207c137533a7efb2e4a0332771b10321bfa1e5bf7a7e9f57aba1"} err="failed to get container status \"4ae4b0264f6e207c137533a7efb2e4a0332771b10321bfa1e5bf7a7e9f57aba1\": rpc error: code = NotFound desc = could not find container \"4ae4b0264f6e207c137533a7efb2e4a0332771b10321bfa1e5bf7a7e9f57aba1\": container with ID starting with 4ae4b0264f6e207c137533a7efb2e4a0332771b10321bfa1e5bf7a7e9f57aba1 not found: ID does not exist"
Oct 30 22:20:53.938181 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:53.938145    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c586c"]
Oct 30 22:20:53.941606 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:53.941582    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c586c"]
Oct 30 22:20:53.961631 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:53.961607    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bef575f8-5sfmq"
Oct 30 22:20:54.141553 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:54.134959    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4402ea44-8969-45e8-8037-593633026e9e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4402ea44-8969-45e8-8037-593633026e9e-maas-api-openshift-ca\") pod \"4402ea44-8969-45e8-8037-593633026e9e\" (UID: \"4402ea44-8969-45e8-8037-593633026e9e\") "
Oct 30 22:20:54.141553 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:54.135092    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4402ea44-8969-45e8-8037-593633026e9e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4402ea44-8969-45e8-8037-593633026e9e-maas-api-xks-ca\") pod \"4402ea44-8969-45e8-8037-593633026e9e\" (UID: \"4402ea44-8969-45e8-8037-593633026e9e\") "
Oct 30 22:20:54.141553 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:54.135261    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4402ea44-8969-45e8-8037-593633026e9e-kube-api-access-7vc25\" (UniqueName: \"kubernetes.io/projected/4402ea44-8969-45e8-8037-593633026e9e-kube-api-access-7vc25\") pod \"4402ea44-8969-45e8-8037-593633026e9e\" (UID: \"4402ea44-8969-45e8-8037-593633026e9e\") "
Oct 30 22:20:54.141553 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:54.136336    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4402ea44-8969-45e8-8037-593633026e9e-maas-api-openshift-ca" pod "4402ea44-8969-45e8-8037-593633026e9e" (UID: "4402ea44-8969-45e8-8037-593633026e9e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:20:54.141553 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:54.139958    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4402ea44-8969-45e8-8037-593633026e9e-maas-api-openshift-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:20:54.148322 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:54.146987    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4402ea44-8969-45e8-8037-593633026e9e-maas-api-xks-ca" pod "4402ea44-8969-45e8-8037-593633026e9e" (UID: "4402ea44-8969-45e8-8037-593633026e9e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:20:54.148322 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:54.148195    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4402ea44-8969-45e8-8037-593633026e9e-kube-api-access-7vc25" pod "4402ea44-8969-45e8-8037-593633026e9e" (UID: "4402ea44-8969-45e8-8037-593633026e9e"). InnerVolumeSpecName "kube-api-access-7vc25". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:20:54.241655 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:54.241571    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4402ea44-8969-45e8-8037-593633026e9e-maas-api-xks-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:20:54.241655 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:54.241603    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vc25\" (UniqueName: \"kubernetes.io/projected/4402ea44-8969-45e8-8037-593633026e9e-kube-api-access-7vc25\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:20:54.380220 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:54.380190    5061 patch_prober.go:29] interesting pod/e2e-mt-778d39d3-openshift-default-bff74755c-k5knf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body=
Oct 30 22:20:54.380380 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:54.380240    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-k5knf" podUID="3489262b-293e-4dba-8bb2-efa4333ab66d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused"
Oct 30 22:20:54.540959 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:54.540880    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc212ddd-b71e-44c3-88f2-f7c0936bf7d4" path="/var/lib/kubelet/pods/dc212ddd-b71e-44c3-88f2-f7c0936bf7d4/volumes"
Oct 30 22:20:54.667596 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:20:54.667560    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 22:20:54.765064 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:54.765035    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bef575f8-5sfmq"
Oct 30 22:20:54.765501 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:54.765078    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bef575f8-5sfmq" event={"ID":"4402ea44-8969-45e8-8037-593633026e9e","Type":"ContainerDied","Data":"81340a80326ea9a32a2d32f1bc815ce55fd36321ce0d472bee10f8049d97b333"}
Oct 30 22:20:54.765501 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:54.765122    5061 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="81340a80326ea9a32a2d32f1bc815ce55fd36321ce0d472bee10f8049d97b333"
Oct 30 22:20:55.379932 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:55.379901    5061 patch_prober.go:29] interesting pod/e2e-mt-778d39d3-openshift-default-bff74755c-k5knf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body=
Oct 30 22:20:55.380113 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:55.379947    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-k5knf" podUID="3489262b-293e-4dba-8bb2-efa4333ab66d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused"
Oct 30 22:20:56.380892 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:56.380855    5061 patch_prober.go:29] interesting pod/e2e-mt-778d39d3-openshift-default-bff74755c-k5knf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body=
Oct 30 22:20:56.381367 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:56.380907    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-k5knf" podUID="3489262b-293e-4dba-8bb2-efa4333ab66d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused"
Oct 30 22:20:56.506073 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:56.506036    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-0ce563-f9b586484-rg56t" podUID="92710489-bfc6-49db-b02d-4519d1b5f3c9" containerName="payload-processing" containerID="cri-o://b1c8373c5f3cea31ee798be283c344de548e284ddf8973316cd5184ac2443624" gracePeriod=25
Oct 30 22:20:56.603494 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:56.603460    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0ce563-6b8c6c95bc-fsrrg" podUID="74b993cb-2ab2-4457-8885-a2ee906a351f" containerName="payload-pre-processing" containerID="cri-o://eb147f6cf6260f99ff52e8729f7c9d9d9cdd4aa0f870a000a20859a65551fbe1" gracePeriod=25
Oct 30 22:20:56.782002 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:56.781968    5061 generic.go:356] "Generic (PLEG): container finished" podID="92710489-bfc6-49db-b02d-4519d1b5f3c9" containerID="b1c8373c5f3cea31ee798be283c344de548e284ddf8973316cd5184ac2443624" exitCode=0
Oct 30 22:20:56.782186 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:56.782053    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-0ce563-f9b586484-rg56t" event={"ID":"92710489-bfc6-49db-b02d-4519d1b5f3c9","Type":"ContainerDied","Data":"b1c8373c5f3cea31ee798be283c344de548e284ddf8973316cd5184ac2443624"}
Oct 30 22:20:56.783772 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:56.783747    5061 generic.go:356] "Generic (PLEG): container finished" podID="74b993cb-2ab2-4457-8885-a2ee906a351f" containerID="eb147f6cf6260f99ff52e8729f7c9d9d9cdd4aa0f870a000a20859a65551fbe1" exitCode=0
Oct 30 22:20:56.783900 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:56.783876    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0ce563-6b8c6c95bc-fsrrg" event={"ID":"74b993cb-2ab2-4457-8885-a2ee906a351f","Type":"ContainerDied","Data":"eb147f6cf6260f99ff52e8729f7c9d9d9cdd4aa0f870a000a20859a65551fbe1"}
Oct 30 22:20:56.949435 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:56.949373    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-0ce563-f9b586484-rg56t"
Oct 30 22:20:56.972445 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:56.971484    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92710489-bfc6-49db-b02d-4519d1b5f3c9-kube-api-access-4ws74\" (UniqueName: \"kubernetes.io/projected/92710489-bfc6-49db-b02d-4519d1b5f3c9-kube-api-access-4ws74\") pod \"92710489-bfc6-49db-b02d-4519d1b5f3c9\" (UID: \"92710489-bfc6-49db-b02d-4519d1b5f3c9\") "
Oct 30 22:20:56.972445 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:56.971598    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92710489-bfc6-49db-b02d-4519d1b5f3c9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92710489-bfc6-49db-b02d-4519d1b5f3c9-plugins-config-volume\") pod \"92710489-bfc6-49db-b02d-4519d1b5f3c9\" (UID: \"92710489-bfc6-49db-b02d-4519d1b5f3c9\") "
Oct 30 22:20:56.972832 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:56.972696    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92710489-bfc6-49db-b02d-4519d1b5f3c9-plugins-config-volume" pod "92710489-bfc6-49db-b02d-4519d1b5f3c9" (UID: "92710489-bfc6-49db-b02d-4519d1b5f3c9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:20:56.975883 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:56.975855    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92710489-bfc6-49db-b02d-4519d1b5f3c9-kube-api-access-4ws74" pod "92710489-bfc6-49db-b02d-4519d1b5f3c9" (UID: "92710489-bfc6-49db-b02d-4519d1b5f3c9"). InnerVolumeSpecName "kube-api-access-4ws74". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:20:56.985575 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:56.985553    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0ce563-6b8c6c95bc-fsrrg"
Oct 30 22:20:57.072979 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:57.072949    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74b993cb-2ab2-4457-8885-a2ee906a351f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74b993cb-2ab2-4457-8885-a2ee906a351f-plugins-config-volume\") pod \"74b993cb-2ab2-4457-8885-a2ee906a351f\" (UID: \"74b993cb-2ab2-4457-8885-a2ee906a351f\") "
Oct 30 22:20:57.073157 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:57.073108    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74b993cb-2ab2-4457-8885-a2ee906a351f-kube-api-access-6mhxx\" (UniqueName: \"kubernetes.io/projected/74b993cb-2ab2-4457-8885-a2ee906a351f-kube-api-access-6mhxx\") pod \"74b993cb-2ab2-4457-8885-a2ee906a351f\" (UID: \"74b993cb-2ab2-4457-8885-a2ee906a351f\") "
Oct 30 22:20:57.073410 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:57.073382    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74b993cb-2ab2-4457-8885-a2ee906a351f-plugins-config-volume" pod "74b993cb-2ab2-4457-8885-a2ee906a351f" (UID: "74b993cb-2ab2-4457-8885-a2ee906a351f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:20:57.073667 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:57.073645    5061 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74b993cb-2ab2-4457-8885-a2ee906a351f-plugins-config-volume\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:20:57.073787 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:57.073668    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ws74\" (UniqueName: \"kubernetes.io/projected/92710489-bfc6-49db-b02d-4519d1b5f3c9-kube-api-access-4ws74\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:20:57.073787 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:57.073685    5061 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92710489-bfc6-49db-b02d-4519d1b5f3c9-plugins-config-volume\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:20:57.079710 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:57.079680    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74b993cb-2ab2-4457-8885-a2ee906a351f-kube-api-access-6mhxx" pod "74b993cb-2ab2-4457-8885-a2ee906a351f" (UID: "74b993cb-2ab2-4457-8885-a2ee906a351f"). InnerVolumeSpecName "kube-api-access-6mhxx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:20:57.175165 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:57.175123    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6mhxx\" (UniqueName: \"kubernetes.io/projected/74b993cb-2ab2-4457-8885-a2ee906a351f-kube-api-access-6mhxx\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:20:57.381516 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:57.381177    5061 patch_prober.go:29] interesting pod/e2e-mt-778d39d3-openshift-default-bff74755c-k5knf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body=
Oct 30 22:20:57.381516 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:57.381236    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-k5knf" podUID="3489262b-293e-4dba-8bb2-efa4333ab66d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused"
Oct 30 22:20:57.486613 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:57.486576    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-778d39d3-544ffbcf87-xvhr4"]
Oct 30 22:20:57.794792 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:57.794760    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-0ce563-f9b586484-rg56t"
Oct 30 22:20:57.794792 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:57.794780    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-0ce563-f9b586484-rg56t" event={"ID":"92710489-bfc6-49db-b02d-4519d1b5f3c9","Type":"ContainerDied","Data":"1c8a01da12ce0ac09e0b982903e4ad718593a026ba4a3eda09e64def9a9268cb"}
Oct 30 22:20:57.795016 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:57.794838    5061 scope.go:122] "RemoveContainer" containerID="b1c8373c5f3cea31ee798be283c344de548e284ddf8973316cd5184ac2443624"
Oct 30 22:20:57.796676 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:57.796658    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0ce563-6b8c6c95bc-fsrrg"
Oct 30 22:20:57.796801 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:57.796657    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0ce563-6b8c6c95bc-fsrrg" event={"ID":"74b993cb-2ab2-4457-8885-a2ee906a351f","Type":"ContainerDied","Data":"68e4f4f2ac1cdfcf225de3574deef114ff450f5ab6e6bfb42ced8015deae27d7"}
Oct 30 22:20:57.821616 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:57.820588    5061 scope.go:122] "RemoveContainer" containerID="eb147f6cf6260f99ff52e8729f7c9d9d9cdd4aa0f870a000a20859a65551fbe1"
Oct 30 22:20:58.058862 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:58.058777    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0ce563-6b8c6c95bc-fsrrg"]
Oct 30 22:20:58.074567 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:58.074539    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0ce563-6b8c6c95bc-fsrrg"]
Oct 30 22:20:58.130264 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:58.130204    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0ce563-f9b586484-rg56t"]
Oct 30 22:20:58.141275 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:58.141246    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0ce563-f9b586484-rg56t"]
Oct 30 22:20:58.189046 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:58.189006    5061 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-c586c container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.116:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:20:58.189210 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:58.189065    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c586c" podUID="dc212ddd-b71e-44c3-88f2-f7c0936bf7d4" containerName="discovery" probeResult="failure" output="Get \"http://10.132.0.116:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:20:58.380326 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:58.380237    5061 patch_prober.go:29] interesting pod/e2e-mt-778d39d3-openshift-default-bff74755c-k5knf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body=
Oct 30 22:20:58.380326 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:58.380309    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-k5knf" podUID="3489262b-293e-4dba-8bb2-efa4333ab66d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused"
Oct 30 22:20:58.529977 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:58.529944    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74b993cb-2ab2-4457-8885-a2ee906a351f" path="/var/lib/kubelet/pods/74b993cb-2ab2-4457-8885-a2ee906a351f/volumes"
Oct 30 22:20:58.530471 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:58.530455    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92710489-bfc6-49db-b02d-4519d1b5f3c9" path="/var/lib/kubelet/pods/92710489-bfc6-49db-b02d-4519d1b5f3c9/volumes"
Oct 30 22:20:59.751033 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:59.750998    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-k5knf"
Oct 30 22:20:59.751463 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:59.751445    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-k5knf"
Oct 30 22:20:59.753367 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:20:59.753348    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-k5knf"
Oct 30 22:21:02.487183 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:02.487140    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-778d39d3-544ffbcf87-xvhr4" podUID="9a858f67-af17-429f-9b06-71d5a7c3a7bf" containerName="payload-pre-processing" containerID="cri-o://6de424961288f6e231abfcebd00a2f0151285170297fd818e3ad9801a3d1c39e" gracePeriod=25
Oct 30 22:21:02.819495 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:02.819466    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-778d39d3-544ffbcf87-xvhr4"
Oct 30 22:21:02.837956 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:02.837920    5061 generic.go:356] "Generic (PLEG): container finished" podID="9a858f67-af17-429f-9b06-71d5a7c3a7bf" containerID="6de424961288f6e231abfcebd00a2f0151285170297fd818e3ad9801a3d1c39e" exitCode=0
Oct 30 22:21:02.838087 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:02.837965    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-778d39d3-544ffbcf87-xvhr4"
Oct 30 22:21:02.838087 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:02.837976    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-778d39d3-544ffbcf87-xvhr4" event={"ID":"9a858f67-af17-429f-9b06-71d5a7c3a7bf","Type":"ContainerDied","Data":"6de424961288f6e231abfcebd00a2f0151285170297fd818e3ad9801a3d1c39e"}
Oct 30 22:21:02.838087 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:02.838029    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-778d39d3-544ffbcf87-xvhr4" event={"ID":"9a858f67-af17-429f-9b06-71d5a7c3a7bf","Type":"ContainerDied","Data":"4f5c94c3ae6cb8d62ad7a869da53990ff0b55ce385fce3fb4dd83e95596f5047"}
Oct 30 22:21:02.838087 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:02.838059    5061 scope.go:122] "RemoveContainer" containerID="6de424961288f6e231abfcebd00a2f0151285170297fd818e3ad9801a3d1c39e"
Oct 30 22:21:02.848422 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:02.848395    5061 scope.go:122] "RemoveContainer" containerID="6de424961288f6e231abfcebd00a2f0151285170297fd818e3ad9801a3d1c39e"
Oct 30 22:21:02.848708 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:21:02.848689    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6de424961288f6e231abfcebd00a2f0151285170297fd818e3ad9801a3d1c39e\": container with ID starting with 6de424961288f6e231abfcebd00a2f0151285170297fd818e3ad9801a3d1c39e not found: ID does not exist" containerID="6de424961288f6e231abfcebd00a2f0151285170297fd818e3ad9801a3d1c39e"
Oct 30 22:21:02.848775 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:02.848716    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6de424961288f6e231abfcebd00a2f0151285170297fd818e3ad9801a3d1c39e"} err="failed to get container status \"6de424961288f6e231abfcebd00a2f0151285170297fd818e3ad9801a3d1c39e\": rpc error: code = NotFound desc = could not find container \"6de424961288f6e231abfcebd00a2f0151285170297fd818e3ad9801a3d1c39e\": container with ID starting with 6de424961288f6e231abfcebd00a2f0151285170297fd818e3ad9801a3d1c39e not found: ID does not exist"
Oct 30 22:21:02.925355 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:02.925322    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a858f67-af17-429f-9b06-71d5a7c3a7bf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a858f67-af17-429f-9b06-71d5a7c3a7bf-plugins-config-volume\") pod \"9a858f67-af17-429f-9b06-71d5a7c3a7bf\" (UID: \"9a858f67-af17-429f-9b06-71d5a7c3a7bf\") "
Oct 30 22:21:02.925643 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:02.925508    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a858f67-af17-429f-9b06-71d5a7c3a7bf-kube-api-access-6tnm4\" (UniqueName: \"kubernetes.io/projected/9a858f67-af17-429f-9b06-71d5a7c3a7bf-kube-api-access-6tnm4\") pod \"9a858f67-af17-429f-9b06-71d5a7c3a7bf\" (UID: \"9a858f67-af17-429f-9b06-71d5a7c3a7bf\") "
Oct 30 22:21:02.926106 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:02.926061    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a858f67-af17-429f-9b06-71d5a7c3a7bf-plugins-config-volume" pod "9a858f67-af17-429f-9b06-71d5a7c3a7bf" (UID: "9a858f67-af17-429f-9b06-71d5a7c3a7bf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:21:02.930128 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:02.930097    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a858f67-af17-429f-9b06-71d5a7c3a7bf-kube-api-access-6tnm4" pod "9a858f67-af17-429f-9b06-71d5a7c3a7bf" (UID: "9a858f67-af17-429f-9b06-71d5a7c3a7bf"). InnerVolumeSpecName "kube-api-access-6tnm4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:21:03.027804 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:03.027769    5061 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a858f67-af17-429f-9b06-71d5a7c3a7bf-plugins-config-volume\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:21:03.027804 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:03.027798    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6tnm4\" (UniqueName: \"kubernetes.io/projected/9a858f67-af17-429f-9b06-71d5a7c3a7bf-kube-api-access-6tnm4\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:21:03.258135 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:03.258076    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-778d39d3-544ffbcf87-xvhr4"]
Oct 30 22:21:03.263474 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:03.263442    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-778d39d3-544ffbcf87-xvhr4"]
Oct 30 22:21:04.284405 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:04.284374    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-84a3cf-openshift-default-6b47cf679-n6j8d"]
Oct 30 22:21:04.285120 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:04.285097    5061 cpu_manager.go:414] "containerMap: removing container" podUID="92710489-bfc6-49db-b02d-4519d1b5f3c9" containerName="payload-processing"
Oct 30 22:21:04.285120 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:04.285119    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92710489-bfc6-49db-b02d-4519d1b5f3c9" containerName="payload-processing"
Oct 30 22:21:04.285355 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:04.285150    5061 cpu_manager.go:414] "containerMap: removing container" podUID="74b993cb-2ab2-4457-8885-a2ee906a351f" containerName="payload-pre-processing"
Oct 30 22:21:04.285355 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:04.285159    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74b993cb-2ab2-4457-8885-a2ee906a351f" containerName="payload-pre-processing"
Oct 30 22:21:04.285355 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:04.285175    5061 cpu_manager.go:414] "containerMap: removing container" podUID="9a858f67-af17-429f-9b06-71d5a7c3a7bf" containerName="payload-pre-processing"
Oct 30 22:21:04.285355 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:04.285180    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a858f67-af17-429f-9b06-71d5a7c3a7bf" containerName="payload-pre-processing"
Oct 30 22:21:04.285355 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:04.285192    5061 cpu_manager.go:414] "containerMap: removing container" podUID="4402ea44-8969-45e8-8037-593633026e9e" containerName="revoke-keys"
Oct 30 22:21:04.285355 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:04.285196    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4402ea44-8969-45e8-8037-593633026e9e" containerName="revoke-keys"
Oct 30 22:21:04.285355 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:04.285208    5061 cpu_manager.go:414] "containerMap: removing container" podUID="dc212ddd-b71e-44c3-88f2-f7c0936bf7d4" containerName="discovery"
Oct 30 22:21:04.285355 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:04.285212    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc212ddd-b71e-44c3-88f2-f7c0936bf7d4" containerName="discovery"
Oct 30 22:21:04.285355 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:04.285349    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="92710489-bfc6-49db-b02d-4519d1b5f3c9" containerName="payload-processing"
Oct 30 22:21:04.285732 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:04.285368    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="74b993cb-2ab2-4457-8885-a2ee906a351f" containerName="payload-pre-processing"
Oct 30 22:21:04.285732 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:04.285376    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a858f67-af17-429f-9b06-71d5a7c3a7bf" containerName="payload-pre-processing"
Oct 30 22:21:04.285732 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:04.285385    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="4402ea44-8969-45e8-8037-593633026e9e" containerName="revoke-keys"
Oct 30 22:21:04.285732 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:04.285398    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc212ddd-b71e-44c3-88f2-f7c0936bf7d4" containerName="discovery"
Oct 30 22:21:04.290470 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:04.290453    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-84a3cf-openshift-default-6b47cf679-n6j8d"
Oct 30 22:21:04.340131 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:04.340104    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-84a3cf-openshift-default-dockercfg-kh5dj\""
Oct 30 22:21:04.344084 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:04.344052    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8cf03377-68fe-44d3-a000-3a4893ef6b8a-istio-data\") pod \"e2e-inf-b-84a3cf-openshift-default-6b47cf679-n6j8d\" (UID: \"8cf03377-68fe-44d3-a000-3a4893ef6b8a\") " pod="openshift-ingress/e2e-inf-b-84a3cf-openshift-default-6b47cf679-n6j8d"
Oct 30 22:21:04.344194 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:04.344115    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cf03377-68fe-44d3-a000-3a4893ef6b8a-workload-socket\") pod \"e2e-inf-b-84a3cf-openshift-default-6b47cf679-n6j8d\" (UID: \"8cf03377-68fe-44d3-a000-3a4893ef6b8a\") " pod="openshift-ingress/e2e-inf-b-84a3cf-openshift-default-6b47cf679-n6j8d"
Oct 30 22:21:04.344194 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:04.344163    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8cf03377-68fe-44d3-a000-3a4893ef6b8a-istio-podinfo\") pod \"e2e-inf-b-84a3cf-openshift-default-6b47cf679-n6j8d\" (UID: \"8cf03377-68fe-44d3-a000-3a4893ef6b8a\") " pod="openshift-ingress/e2e-inf-b-84a3cf-openshift-default-6b47cf679-n6j8d"
Oct 30 22:21:04.344298 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:04.344204    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqx7z\" (UniqueName: \"kubernetes.io/projected/8cf03377-68fe-44d3-a000-3a4893ef6b8a-kube-api-access-fqx7z\") pod \"e2e-inf-b-84a3cf-openshift-default-6b47cf679-n6j8d\" (UID: \"8cf03377-68fe-44d3-a000-3a4893ef6b8a\") " pod="openshift-ingress/e2e-inf-b-84a3cf-openshift-default-6b47cf679-n6j8d"
Oct 30 22:21:04.344298 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:04.344237    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8cf03377-68fe-44d3-a000-3a4893ef6b8a-istiod-ca-cert\") pod \"e2e-inf-b-84a3cf-openshift-default-6b47cf679-n6j8d\" (UID: \"8cf03377-68fe-44d3-a000-3a4893ef6b8a\") " pod="openshift-ingress/e2e-inf-b-84a3cf-openshift-default-6b47cf679-n6j8d"
Oct 30 22:21:04.344398 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:04.344366    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8cf03377-68fe-44d3-a000-3a4893ef6b8a-istio-envoy\") pod \"e2e-inf-b-84a3cf-openshift-default-6b47cf679-n6j8d\" (UID: \"8cf03377-68fe-44d3-a000-3a4893ef6b8a\") " pod="openshift-ingress/e2e-inf-b-84a3cf-openshift-default-6b47cf679-n6j8d"
Oct 30 22:21:04.344441 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:04.344399    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cf03377-68fe-44d3-a000-3a4893ef6b8a-credential-socket\") pod \"e2e-inf-b-84a3cf-openshift-default-6b47cf679-n6j8d\" (UID: \"8cf03377-68fe-44d3-a000-3a4893ef6b8a\") " pod="openshift-ingress/e2e-inf-b-84a3cf-openshift-default-6b47cf679-n6j8d"
Oct 30 22:21:04.344497 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:04.344482    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8cf03377-68fe-44d3-a000-3a4893ef6b8a-istio-token\") pod \"e2e-inf-b-84a3cf-openshift-default-6b47cf679-n6j8d\" (UID: \"8cf03377-68fe-44d3-a000-3a4893ef6b8a\") " pod="openshift-ingress/e2e-inf-b-84a3cf-openshift-default-6b47cf679-n6j8d"
Oct 30 22:21:04.344560 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:04.344515    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8cf03377-68fe-44d3-a000-3a4893ef6b8a-workload-certs\") pod \"e2e-inf-b-84a3cf-openshift-default-6b47cf679-n6j8d\" (UID: \"8cf03377-68fe-44d3-a000-3a4893ef6b8a\") " pod="openshift-ingress/e2e-inf-b-84a3cf-openshift-default-6b47cf679-n6j8d"
Oct 30 22:21:04.390722 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:04.390690    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-84a3cf-openshift-default-6b47cf679-n6j8d"]
Oct 30 22:21:04.446285 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:04.446256    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8cf03377-68fe-44d3-a000-3a4893ef6b8a-istio-envoy\") pod \"e2e-inf-b-84a3cf-openshift-default-6b47cf679-n6j8d\" (UID: \"8cf03377-68fe-44d3-a000-3a4893ef6b8a\") " pod="openshift-ingress/e2e-inf-b-84a3cf-openshift-default-6b47cf679-n6j8d"
Oct 30 22:21:04.446285 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:04.446289    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cf03377-68fe-44d3-a000-3a4893ef6b8a-credential-socket\") pod \"e2e-inf-b-84a3cf-openshift-default-6b47cf679-n6j8d\" (UID: \"8cf03377-68fe-44d3-a000-3a4893ef6b8a\") " pod="openshift-ingress/e2e-inf-b-84a3cf-openshift-default-6b47cf679-n6j8d"
Oct 30 22:21:04.446497 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:04.446429    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8cf03377-68fe-44d3-a000-3a4893ef6b8a-istio-token\") pod \"e2e-inf-b-84a3cf-openshift-default-6b47cf679-n6j8d\" (UID: \"8cf03377-68fe-44d3-a000-3a4893ef6b8a\") " pod="openshift-ingress/e2e-inf-b-84a3cf-openshift-default-6b47cf679-n6j8d"
Oct 30 22:21:04.446497 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:04.446467    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8cf03377-68fe-44d3-a000-3a4893ef6b8a-workload-certs\") pod \"e2e-inf-b-84a3cf-openshift-default-6b47cf679-n6j8d\" (UID: \"8cf03377-68fe-44d3-a000-3a4893ef6b8a\") " pod="openshift-ingress/e2e-inf-b-84a3cf-openshift-default-6b47cf679-n6j8d"
Oct 30 22:21:04.446643 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:04.446566    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8cf03377-68fe-44d3-a000-3a4893ef6b8a-istio-data\") pod \"e2e-inf-b-84a3cf-openshift-default-6b47cf679-n6j8d\" (UID: \"8cf03377-68fe-44d3-a000-3a4893ef6b8a\") " pod="openshift-ingress/e2e-inf-b-84a3cf-openshift-default-6b47cf679-n6j8d"
Oct 30 22:21:04.446643 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:04.446613    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cf03377-68fe-44d3-a000-3a4893ef6b8a-workload-socket\") pod \"e2e-inf-b-84a3cf-openshift-default-6b47cf679-n6j8d\" (UID: \"8cf03377-68fe-44d3-a000-3a4893ef6b8a\") " pod="openshift-ingress/e2e-inf-b-84a3cf-openshift-default-6b47cf679-n6j8d"
Oct 30 22:21:04.446754 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:04.446645    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8cf03377-68fe-44d3-a000-3a4893ef6b8a-istio-podinfo\") pod \"e2e-inf-b-84a3cf-openshift-default-6b47cf679-n6j8d\" (UID: \"8cf03377-68fe-44d3-a000-3a4893ef6b8a\") " pod="openshift-ingress/e2e-inf-b-84a3cf-openshift-default-6b47cf679-n6j8d"
Oct 30 22:21:04.446754 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:04.446667    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cf03377-68fe-44d3-a000-3a4893ef6b8a-credential-socket\") pod \"e2e-inf-b-84a3cf-openshift-default-6b47cf679-n6j8d\" (UID: \"8cf03377-68fe-44d3-a000-3a4893ef6b8a\") " pod="openshift-ingress/e2e-inf-b-84a3cf-openshift-default-6b47cf679-n6j8d"
Oct 30 22:21:04.446754 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:04.446700    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqx7z\" (UniqueName: \"kubernetes.io/projected/8cf03377-68fe-44d3-a000-3a4893ef6b8a-kube-api-access-fqx7z\") pod \"e2e-inf-b-84a3cf-openshift-default-6b47cf679-n6j8d\" (UID: \"8cf03377-68fe-44d3-a000-3a4893ef6b8a\") " pod="openshift-ingress/e2e-inf-b-84a3cf-openshift-default-6b47cf679-n6j8d"
Oct 30 22:21:04.446754 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:04.446748    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8cf03377-68fe-44d3-a000-3a4893ef6b8a-istiod-ca-cert\") pod \"e2e-inf-b-84a3cf-openshift-default-6b47cf679-n6j8d\" (UID: \"8cf03377-68fe-44d3-a000-3a4893ef6b8a\") " pod="openshift-ingress/e2e-inf-b-84a3cf-openshift-default-6b47cf679-n6j8d"
Oct 30 22:21:04.446935 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:04.446841    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8cf03377-68fe-44d3-a000-3a4893ef6b8a-workload-certs\") pod \"e2e-inf-b-84a3cf-openshift-default-6b47cf679-n6j8d\" (UID: \"8cf03377-68fe-44d3-a000-3a4893ef6b8a\") " pod="openshift-ingress/e2e-inf-b-84a3cf-openshift-default-6b47cf679-n6j8d"
Oct 30 22:21:04.446990 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:04.446975    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8cf03377-68fe-44d3-a000-3a4893ef6b8a-istio-data\") pod \"e2e-inf-b-84a3cf-openshift-default-6b47cf679-n6j8d\" (UID: \"8cf03377-68fe-44d3-a000-3a4893ef6b8a\") " pod="openshift-ingress/e2e-inf-b-84a3cf-openshift-default-6b47cf679-n6j8d"
Oct 30 22:21:04.447054 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:04.447028    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cf03377-68fe-44d3-a000-3a4893ef6b8a-workload-socket\") pod \"e2e-inf-b-84a3cf-openshift-default-6b47cf679-n6j8d\" (UID: \"8cf03377-68fe-44d3-a000-3a4893ef6b8a\") " pod="openshift-ingress/e2e-inf-b-84a3cf-openshift-default-6b47cf679-n6j8d"
Oct 30 22:21:04.447389 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:04.447370    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8cf03377-68fe-44d3-a000-3a4893ef6b8a-istiod-ca-cert\") pod \"e2e-inf-b-84a3cf-openshift-default-6b47cf679-n6j8d\" (UID: \"8cf03377-68fe-44d3-a000-3a4893ef6b8a\") " pod="openshift-ingress/e2e-inf-b-84a3cf-openshift-default-6b47cf679-n6j8d"
Oct 30 22:21:04.448817 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:04.448795    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8cf03377-68fe-44d3-a000-3a4893ef6b8a-istio-envoy\") pod \"e2e-inf-b-84a3cf-openshift-default-6b47cf679-n6j8d\" (UID: \"8cf03377-68fe-44d3-a000-3a4893ef6b8a\") " pod="openshift-ingress/e2e-inf-b-84a3cf-openshift-default-6b47cf679-n6j8d"
Oct 30 22:21:04.449066 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:04.449050    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8cf03377-68fe-44d3-a000-3a4893ef6b8a-istio-podinfo\") pod \"e2e-inf-b-84a3cf-openshift-default-6b47cf679-n6j8d\" (UID: \"8cf03377-68fe-44d3-a000-3a4893ef6b8a\") " pod="openshift-ingress/e2e-inf-b-84a3cf-openshift-default-6b47cf679-n6j8d"
Oct 30 22:21:04.500291 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:04.500266    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8cf03377-68fe-44d3-a000-3a4893ef6b8a-istio-token\") pod \"e2e-inf-b-84a3cf-openshift-default-6b47cf679-n6j8d\" (UID: \"8cf03377-68fe-44d3-a000-3a4893ef6b8a\") " pod="openshift-ingress/e2e-inf-b-84a3cf-openshift-default-6b47cf679-n6j8d"
Oct 30 22:21:04.500521 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:04.500499    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqx7z\" (UniqueName: \"kubernetes.io/projected/8cf03377-68fe-44d3-a000-3a4893ef6b8a-kube-api-access-fqx7z\") pod \"e2e-inf-b-84a3cf-openshift-default-6b47cf679-n6j8d\" (UID: \"8cf03377-68fe-44d3-a000-3a4893ef6b8a\") " pod="openshift-ingress/e2e-inf-b-84a3cf-openshift-default-6b47cf679-n6j8d"
Oct 30 22:21:04.566881 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:04.566785    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a858f67-af17-429f-9b06-71d5a7c3a7bf" path="/var/lib/kubelet/pods/9a858f67-af17-429f-9b06-71d5a7c3a7bf/volumes"
Oct 30 22:21:04.604370 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:04.604332    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-84a3cf-openshift-default-6b47cf679-n6j8d"
Oct 30 22:21:05.027711 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:05.027633    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-84a3cf-openshift-default-6b47cf679-n6j8d"]
Oct 30 22:21:05.059928 ip-10-0-130-60 kubenswrapper[5061]: W1030 22:21:05.059897    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cf03377_68fe_44d3_a000_3a4893ef6b8a.slice/crio-2e79d8da5317b01f9d36432d61de4883da77d24b5534c945073b87333d21596a WatchSource:0}: Error finding container 2e79d8da5317b01f9d36432d61de4883da77d24b5534c945073b87333d21596a: Status 404 returned error can't find the container with id 2e79d8da5317b01f9d36432d61de4883da77d24b5534c945073b87333d21596a
Oct 30 22:21:05.062539 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:05.062496    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:21:05.062617 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:05.062607    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:21:05.062653 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:05.062640    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:21:05.867465 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:05.867424    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-84a3cf-openshift-default-6b47cf679-n6j8d" event={"ID":"8cf03377-68fe-44d3-a000-3a4893ef6b8a","Type":"ContainerStarted","Data":"3b1e4f5beb386b9fd37021e9ec9fb3dc409435a41568715df942b5d8c5e1b869"}
Oct 30 22:21:05.867960 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:05.867479    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-84a3cf-openshift-default-6b47cf679-n6j8d" event={"ID":"8cf03377-68fe-44d3-a000-3a4893ef6b8a","Type":"ContainerStarted","Data":"2e79d8da5317b01f9d36432d61de4883da77d24b5534c945073b87333d21596a"}
Oct 30 22:21:06.029976 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:06.029918    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-84a3cf-openshift-default-6b47cf679-n6j8d" podStartSLOduration=2.029902553 podStartE2EDuration="2.029902553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:21:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:21:06.027189489 +0000 UTC m=+2319.639959267" watchObservedRunningTime="2026-10-30 22:21:06.029902553 +0000 UTC m=+2319.642672312"
Oct 30 22:21:06.152519 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:06.152399    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-649qh"]
Oct 30 22:21:06.166374 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:06.166335    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-649qh"
Oct 30 22:21:06.241258 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:06.241227    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-649qh"]
Oct 30 22:21:06.286926 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:06.286894    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/598138a5-2e71-4727-9332-3c604144f0ef-istio-data\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-649qh\" (UID: \"598138a5-2e71-4727-9332-3c604144f0ef\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-649qh"
Oct 30 22:21:06.286926 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:06.286929    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/598138a5-2e71-4727-9332-3c604144f0ef-istiod-ca-cert\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-649qh\" (UID: \"598138a5-2e71-4727-9332-3c604144f0ef\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-649qh"
Oct 30 22:21:06.287150 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:06.286955    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/598138a5-2e71-4727-9332-3c604144f0ef-istio-token\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-649qh\" (UID: \"598138a5-2e71-4727-9332-3c604144f0ef\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-649qh"
Oct 30 22:21:06.287150 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:06.287026    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/598138a5-2e71-4727-9332-3c604144f0ef-workload-socket\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-649qh\" (UID: \"598138a5-2e71-4727-9332-3c604144f0ef\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-649qh"
Oct 30 22:21:06.287233 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:06.287151    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/598138a5-2e71-4727-9332-3c604144f0ef-istio-podinfo\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-649qh\" (UID: \"598138a5-2e71-4727-9332-3c604144f0ef\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-649qh"
Oct 30 22:21:06.287233 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:06.287210    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zkwn\" (UniqueName: \"kubernetes.io/projected/598138a5-2e71-4727-9332-3c604144f0ef-kube-api-access-6zkwn\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-649qh\" (UID: \"598138a5-2e71-4727-9332-3c604144f0ef\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-649qh"
Oct 30 22:21:06.287354 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:06.287332    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/598138a5-2e71-4727-9332-3c604144f0ef-credential-socket\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-649qh\" (UID: \"598138a5-2e71-4727-9332-3c604144f0ef\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-649qh"
Oct 30 22:21:06.287415 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:06.287398    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/598138a5-2e71-4727-9332-3c604144f0ef-istio-envoy\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-649qh\" (UID: \"598138a5-2e71-4727-9332-3c604144f0ef\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-649qh"
Oct 30 22:21:06.287480 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:06.287465    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/598138a5-2e71-4727-9332-3c604144f0ef-workload-certs\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-649qh\" (UID: \"598138a5-2e71-4727-9332-3c604144f0ef\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-649qh"
Oct 30 22:21:06.389069 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:06.389028    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/598138a5-2e71-4727-9332-3c604144f0ef-istio-envoy\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-649qh\" (UID: \"598138a5-2e71-4727-9332-3c604144f0ef\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-649qh"
Oct 30 22:21:06.389261 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:06.389091    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/598138a5-2e71-4727-9332-3c604144f0ef-workload-certs\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-649qh\" (UID: \"598138a5-2e71-4727-9332-3c604144f0ef\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-649qh"
Oct 30 22:21:06.389341 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:06.389263    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/598138a5-2e71-4727-9332-3c604144f0ef-istio-data\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-649qh\" (UID: \"598138a5-2e71-4727-9332-3c604144f0ef\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-649qh"
Oct 30 22:21:06.389341 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:06.389293    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/598138a5-2e71-4727-9332-3c604144f0ef-istiod-ca-cert\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-649qh\" (UID: \"598138a5-2e71-4727-9332-3c604144f0ef\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-649qh"
Oct 30 22:21:06.389341 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:06.389329    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/598138a5-2e71-4727-9332-3c604144f0ef-istio-token\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-649qh\" (UID: \"598138a5-2e71-4727-9332-3c604144f0ef\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-649qh"
Oct 30 22:21:06.389506 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:06.389360    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/598138a5-2e71-4727-9332-3c604144f0ef-workload-socket\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-649qh\" (UID: \"598138a5-2e71-4727-9332-3c604144f0ef\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-649qh"
Oct 30 22:21:06.389506 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:06.389390    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/598138a5-2e71-4727-9332-3c604144f0ef-istio-podinfo\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-649qh\" (UID: \"598138a5-2e71-4727-9332-3c604144f0ef\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-649qh"
Oct 30 22:21:06.389506 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:06.389433    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zkwn\" (UniqueName: \"kubernetes.io/projected/598138a5-2e71-4727-9332-3c604144f0ef-kube-api-access-6zkwn\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-649qh\" (UID: \"598138a5-2e71-4727-9332-3c604144f0ef\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-649qh"
Oct 30 22:21:06.389506 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:06.389489    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/598138a5-2e71-4727-9332-3c604144f0ef-credential-socket\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-649qh\" (UID: \"598138a5-2e71-4727-9332-3c604144f0ef\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-649qh"
Oct 30 22:21:06.389744 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:06.389569    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/598138a5-2e71-4727-9332-3c604144f0ef-workload-certs\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-649qh\" (UID: \"598138a5-2e71-4727-9332-3c604144f0ef\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-649qh"
Oct 30 22:21:06.389744 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:06.389628    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/598138a5-2e71-4727-9332-3c604144f0ef-istio-data\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-649qh\" (UID: \"598138a5-2e71-4727-9332-3c604144f0ef\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-649qh"
Oct 30 22:21:06.389857 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:06.389742    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/598138a5-2e71-4727-9332-3c604144f0ef-workload-socket\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-649qh\" (UID: \"598138a5-2e71-4727-9332-3c604144f0ef\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-649qh"
Oct 30 22:21:06.389916 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:06.389873    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/598138a5-2e71-4727-9332-3c604144f0ef-credential-socket\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-649qh\" (UID: \"598138a5-2e71-4727-9332-3c604144f0ef\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-649qh"
Oct 30 22:21:06.390022 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:06.390002    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/598138a5-2e71-4727-9332-3c604144f0ef-istiod-ca-cert\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-649qh\" (UID: \"598138a5-2e71-4727-9332-3c604144f0ef\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-649qh"
Oct 30 22:21:06.391517 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:06.391496    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/598138a5-2e71-4727-9332-3c604144f0ef-istio-envoy\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-649qh\" (UID: \"598138a5-2e71-4727-9332-3c604144f0ef\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-649qh"
Oct 30 22:21:06.391867 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:06.391850    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/598138a5-2e71-4727-9332-3c604144f0ef-istio-podinfo\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-649qh\" (UID: \"598138a5-2e71-4727-9332-3c604144f0ef\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-649qh"
Oct 30 22:21:06.443512 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:06.443456    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/598138a5-2e71-4727-9332-3c604144f0ef-istio-token\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-649qh\" (UID: \"598138a5-2e71-4727-9332-3c604144f0ef\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-649qh"
Oct 30 22:21:06.443634 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:06.443619    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zkwn\" (UniqueName: \"kubernetes.io/projected/598138a5-2e71-4727-9332-3c604144f0ef-kube-api-access-6zkwn\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-649qh\" (UID: \"598138a5-2e71-4727-9332-3c604144f0ef\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-649qh"
Oct 30 22:21:06.483771 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:06.483745    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-649qh"
Oct 30 22:21:06.605027 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:06.604957    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-84a3cf-openshift-default-6b47cf679-n6j8d"
Oct 30 22:21:06.611758 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:06.611732    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-84a3cf-openshift-default-6b47cf679-n6j8d"
Oct 30 22:21:06.878593 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:06.878563    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-84a3cf-openshift-default-6b47cf679-n6j8d"
Oct 30 22:21:06.879591 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:06.879573    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-84a3cf-openshift-default-6b47cf679-n6j8d"
Oct 30 22:21:06.918814 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:06.918766    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-649qh"]
Oct 30 22:21:06.918931 ip-10-0-130-60 kubenswrapper[5061]: W1030 22:21:06.918874    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod598138a5_2e71_4727_9332_3c604144f0ef.slice/crio-0684a5651d0c6871a33081643ba48d3a9af9bbcab8136b520a37de84d316affd WatchSource:0}: Error finding container 0684a5651d0c6871a33081643ba48d3a9af9bbcab8136b520a37de84d316affd: Status 404 returned error can't find the container with id 0684a5651d0c6871a33081643ba48d3a9af9bbcab8136b520a37de84d316affd
Oct 30 22:21:06.921890 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:06.921852    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:21:06.921975 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:06.921933    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:21:06.922017 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:06.921980    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:21:07.887489 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:07.887447    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-649qh" event={"ID":"598138a5-2e71-4727-9332-3c604144f0ef","Type":"ContainerStarted","Data":"b38d0702c46a0f99ecacb9cbe7ed469a5bd18364eb5845fc32519c06825cc329"}
Oct 30 22:21:07.887964 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:07.887566    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-649qh" event={"ID":"598138a5-2e71-4727-9332-3c604144f0ef","Type":"ContainerStarted","Data":"0684a5651d0c6871a33081643ba48d3a9af9bbcab8136b520a37de84d316affd"}
Oct 30 22:21:08.003892 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:08.003839    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-649qh" podStartSLOduration=2.003825461 podStartE2EDuration="2.003825461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:21:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:21:08.000660727 +0000 UTC m=+2321.613430523" watchObservedRunningTime="2026-10-30 22:21:08.003825461 +0000 UTC m=+2321.616595218"
Oct 30 22:21:08.484705 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:08.484673    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-649qh"
Oct 30 22:21:08.486662 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:08.486636    5061 patch_prober.go:29] interesting pod/e2e-mt-778d39d3-openshift-default-bff74755c-649qh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body=
Oct 30 22:21:08.486770 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:08.486681    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-649qh" podUID="598138a5-2e71-4727-9332-3c604144f0ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused"
Oct 30 22:21:09.485791 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:09.484472    5061 patch_prober.go:29] interesting pod/e2e-mt-778d39d3-openshift-default-bff74755c-649qh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body=
Oct 30 22:21:09.485791 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:09.484548    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-649qh" podUID="598138a5-2e71-4727-9332-3c604144f0ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused"
Oct 30 22:21:10.484251 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:10.484215    5061 patch_prober.go:29] interesting pod/e2e-mt-778d39d3-openshift-default-bff74755c-649qh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body=
Oct 30 22:21:10.484409 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:10.484270    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-649qh" podUID="598138a5-2e71-4727-9332-3c604144f0ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused"
Oct 30 22:21:11.484703 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:11.484667    5061 patch_prober.go:29] interesting pod/e2e-mt-778d39d3-openshift-default-bff74755c-649qh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body=
Oct 30 22:21:11.485110 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:11.484718    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-649qh" podUID="598138a5-2e71-4727-9332-3c604144f0ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused"
Oct 30 22:21:11.993407 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:11.990942    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-chgcq"]
Oct 30 22:21:11.993407 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:11.991120    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-chgcq" podUID="147f4f98-ac6a-4b58-8750-a8410dd6324e" containerName="istio-proxy" containerID="cri-o://ec9430b1c8a6e0956573be159466e09daa706f3ceba7cafd80697c5747f4b8fc" gracePeriod=30
Oct 30 22:21:12.487795 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:12.487743    5061 patch_prober.go:29] interesting pod/e2e-mt-778d39d3-openshift-default-bff74755c-649qh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body=
Oct 30 22:21:12.488667 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:12.488634    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-649qh" podUID="598138a5-2e71-4727-9332-3c604144f0ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused"
Oct 30 22:21:12.503593 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:12.503560    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4215ed-ph8sn"]
Oct 30 22:21:12.548191 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:12.548164    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4215ed-ph8sn"]
Oct 30 22:21:12.804162 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:12.804130    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-84a3cf-6c95488bd4-cjmzh"]
Oct 30 22:21:12.835224 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:12.835194    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-84a3cf-6c95488bd4-cjmzh"
Oct 30 22:21:12.855048 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:12.854332    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bzcs\" (UniqueName: \"kubernetes.io/projected/7e098fc6-5863-4aa7-a134-f70fc5380a0d-kube-api-access-9bzcs\") pod \"payload-processing-e2e-inf-b-84a3cf-6c95488bd4-cjmzh\" (UID: \"7e098fc6-5863-4aa7-a134-f70fc5380a0d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-84a3cf-6c95488bd4-cjmzh"
Oct 30 22:21:12.855048 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:12.854380    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e098fc6-5863-4aa7-a134-f70fc5380a0d-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-84a3cf-6c95488bd4-cjmzh\" (UID: \"7e098fc6-5863-4aa7-a134-f70fc5380a0d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-84a3cf-6c95488bd4-cjmzh"
Oct 30 22:21:12.871881 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:12.871846    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-84a3cf-6c95488bd4-cjmzh"]
Oct 30 22:21:12.891411 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:12.891382    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-84a3cf\""
Oct 30 22:21:12.891854 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:12.891836    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-84a3cf-dockercfg-4pv7w\""
Oct 30 22:21:13.003546 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:12.994071    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bzcs\" (UniqueName: \"kubernetes.io/projected/7e098fc6-5863-4aa7-a134-f70fc5380a0d-kube-api-access-9bzcs\") pod \"payload-processing-e2e-inf-b-84a3cf-6c95488bd4-cjmzh\" (UID: \"7e098fc6-5863-4aa7-a134-f70fc5380a0d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-84a3cf-6c95488bd4-cjmzh"
Oct 30 22:21:13.003546 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:12.995697    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e098fc6-5863-4aa7-a134-f70fc5380a0d-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-84a3cf-6c95488bd4-cjmzh\" (UID: \"7e098fc6-5863-4aa7-a134-f70fc5380a0d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-84a3cf-6c95488bd4-cjmzh"
Oct 30 22:21:13.003546 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:12.996694    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e098fc6-5863-4aa7-a134-f70fc5380a0d-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-84a3cf-6c95488bd4-cjmzh\" (UID: \"7e098fc6-5863-4aa7-a134-f70fc5380a0d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-84a3cf-6c95488bd4-cjmzh"
Oct 30 22:21:13.069695 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:13.051282    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-84a3cf-6cb65fb6cf-r64nq"]
Oct 30 22:21:13.069695 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:13.055249    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bzcs\" (UniqueName: \"kubernetes.io/projected/7e098fc6-5863-4aa7-a134-f70fc5380a0d-kube-api-access-9bzcs\") pod \"payload-processing-e2e-inf-b-84a3cf-6c95488bd4-cjmzh\" (UID: \"7e098fc6-5863-4aa7-a134-f70fc5380a0d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-84a3cf-6c95488bd4-cjmzh"
Oct 30 22:21:13.069695 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:13.056699    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-84a3cf-6cb65fb6cf-r64nq"
Oct 30 22:21:13.079437 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:13.079248    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-84a3cf-6cb65fb6cf-r64nq"]
Oct 30 22:21:13.156309 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:13.156273    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63cdc87a-9959-41f0-b5c8-1df23cf0cda4-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-84a3cf-6cb65fb6cf-r64nq\" (UID: \"63cdc87a-9959-41f0-b5c8-1df23cf0cda4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-84a3cf-6cb65fb6cf-r64nq"
Oct 30 22:21:13.156512 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:13.156342    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dnvm\" (UniqueName: \"kubernetes.io/projected/63cdc87a-9959-41f0-b5c8-1df23cf0cda4-kube-api-access-4dnvm\") pod \"payload-pre-processing-e2e-inf-b-84a3cf-6cb65fb6cf-r64nq\" (UID: \"63cdc87a-9959-41f0-b5c8-1df23cf0cda4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-84a3cf-6cb65fb6cf-r64nq"
Oct 30 22:21:13.178280 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:13.177991    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-84a3cf-6c95488bd4-cjmzh"
Oct 30 22:21:13.266014 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:13.265697    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63cdc87a-9959-41f0-b5c8-1df23cf0cda4-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-84a3cf-6cb65fb6cf-r64nq\" (UID: \"63cdc87a-9959-41f0-b5c8-1df23cf0cda4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-84a3cf-6cb65fb6cf-r64nq"
Oct 30 22:21:13.266014 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:13.265799    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dnvm\" (UniqueName: \"kubernetes.io/projected/63cdc87a-9959-41f0-b5c8-1df23cf0cda4-kube-api-access-4dnvm\") pod \"payload-pre-processing-e2e-inf-b-84a3cf-6cb65fb6cf-r64nq\" (UID: \"63cdc87a-9959-41f0-b5c8-1df23cf0cda4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-84a3cf-6cb65fb6cf-r64nq"
Oct 30 22:21:13.269483 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:13.269446    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63cdc87a-9959-41f0-b5c8-1df23cf0cda4-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-84a3cf-6cb65fb6cf-r64nq\" (UID: \"63cdc87a-9959-41f0-b5c8-1df23cf0cda4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-84a3cf-6cb65fb6cf-r64nq"
Oct 30 22:21:13.327414 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:13.327325    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dnvm\" (UniqueName: \"kubernetes.io/projected/63cdc87a-9959-41f0-b5c8-1df23cf0cda4-kube-api-access-4dnvm\") pod \"payload-pre-processing-e2e-inf-b-84a3cf-6cb65fb6cf-r64nq\" (UID: \"63cdc87a-9959-41f0-b5c8-1df23cf0cda4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-84a3cf-6cb65fb6cf-r64nq"
Oct 30 22:21:13.399689 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:13.399658    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-84a3cf-6cb65fb6cf-r64nq"
Oct 30 22:21:13.484776 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:13.484146    5061 patch_prober.go:29] interesting pod/e2e-mt-778d39d3-openshift-default-bff74755c-649qh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body=
Oct 30 22:21:13.484776 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:13.484205    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-649qh" podUID="598138a5-2e71-4727-9332-3c604144f0ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused"
Oct 30 22:21:13.597380 ip-10-0-130-60 kubenswrapper[5061]: W1030 22:21:13.597347    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e098fc6_5863_4aa7_a134_f70fc5380a0d.slice/crio-3cfd0325bd8c88c7e29e4d19c5fdf4578861a91b5a5049ec9d680babe24487cc WatchSource:0}: Error finding container 3cfd0325bd8c88c7e29e4d19c5fdf4578861a91b5a5049ec9d680babe24487cc: Status 404 returned error can't find the container with id 3cfd0325bd8c88c7e29e4d19c5fdf4578861a91b5a5049ec9d680babe24487cc
Oct 30 22:21:13.597840 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:13.597817    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-84a3cf-6c95488bd4-cjmzh"]
Oct 30 22:21:13.777298 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:13.777264    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-84a3cf-6cb65fb6cf-r64nq"]
Oct 30 22:21:13.777846 ip-10-0-130-60 kubenswrapper[5061]: W1030 22:21:13.777808    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63cdc87a_9959_41f0_b5c8_1df23cf0cda4.slice/crio-1fe812826553827988493bc7739b24f93a49da3c344e6754c78022c668688d64 WatchSource:0}: Error finding container 1fe812826553827988493bc7739b24f93a49da3c344e6754c78022c668688d64: Status 404 returned error can't find the container with id 1fe812826553827988493bc7739b24f93a49da3c344e6754c78022c668688d64
Oct 30 22:21:14.080982 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:14.080950    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-84a3cf-6cb65fb6cf-r64nq" event={"ID":"63cdc87a-9959-41f0-b5c8-1df23cf0cda4","Type":"ContainerStarted","Data":"c28cfbfabc772b3ed0af12640598e35d422138fe420beef8bacf1a16879d1bcb"}
Oct 30 22:21:14.081172 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:14.081003    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-84a3cf-6cb65fb6cf-r64nq" event={"ID":"63cdc87a-9959-41f0-b5c8-1df23cf0cda4","Type":"ContainerStarted","Data":"1fe812826553827988493bc7739b24f93a49da3c344e6754c78022c668688d64"}
Oct 30 22:21:14.081172 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:14.081066    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-84a3cf-6cb65fb6cf-r64nq"
Oct 30 22:21:14.082712 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:14.082683    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-84a3cf-6c95488bd4-cjmzh" event={"ID":"7e098fc6-5863-4aa7-a134-f70fc5380a0d","Type":"ContainerStarted","Data":"37ba9b5cd8f7b071d28c86233af2bdb00bb896124c04340e88670ad1642596b3"}
Oct 30 22:21:14.082818 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:14.082729    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-84a3cf-6c95488bd4-cjmzh" event={"ID":"7e098fc6-5863-4aa7-a134-f70fc5380a0d","Type":"ContainerStarted","Data":"3cfd0325bd8c88c7e29e4d19c5fdf4578861a91b5a5049ec9d680babe24487cc"}
Oct 30 22:21:14.082818 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:14.082756    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-84a3cf-6c95488bd4-cjmzh"
Oct 30 22:21:14.194062 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:14.193976    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-84a3cf-6cb65fb6cf-r64nq" podStartSLOduration=2.193963961 podStartE2EDuration="2.193963961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:21:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:21:14.193287641 +0000 UTC m=+2327.806057399" watchObservedRunningTime="2026-10-30 22:21:14.193963961 +0000 UTC m=+2327.806733718"
Oct 30 22:21:14.309641 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:14.309570    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-84a3cf-6c95488bd4-cjmzh" podStartSLOduration=2.309551278 podStartE2EDuration="2.309551278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:21:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:21:14.305370591 +0000 UTC m=+2327.918140349" watchObservedRunningTime="2026-10-30 22:21:14.309551278 +0000 UTC m=+2327.922321031"
Oct 30 22:21:14.489152 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:14.489081    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-649qh"
Oct 30 22:21:14.489398 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:14.489377    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-649qh"
Oct 30 22:21:14.490246 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:14.490227    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-649qh"
Oct 30 22:21:14.529760 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:14.529730    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40263768-14ff-472a-b17c-35bbf52809a3" path="/var/lib/kubelet/pods/40263768-14ff-472a-b17c-35bbf52809a3/volumes"
Oct 30 22:21:17.262581 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:17.262493    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-chgcq"
Oct 30 22:21:17.321475 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:17.321449    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/147f4f98-ac6a-4b58-8750-a8410dd6324e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/147f4f98-ac6a-4b58-8750-a8410dd6324e-workload-certs\") pod \"147f4f98-ac6a-4b58-8750-a8410dd6324e\" (UID: \"147f4f98-ac6a-4b58-8750-a8410dd6324e\") "
Oct 30 22:21:17.321665 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:17.321499    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/147f4f98-ac6a-4b58-8750-a8410dd6324e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/147f4f98-ac6a-4b58-8750-a8410dd6324e-istio-data\") pod \"147f4f98-ac6a-4b58-8750-a8410dd6324e\" (UID: \"147f4f98-ac6a-4b58-8750-a8410dd6324e\") "
Oct 30 22:21:17.321665 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:17.321588    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/147f4f98-ac6a-4b58-8750-a8410dd6324e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/147f4f98-ac6a-4b58-8750-a8410dd6324e-credential-socket\") pod \"147f4f98-ac6a-4b58-8750-a8410dd6324e\" (UID: \"147f4f98-ac6a-4b58-8750-a8410dd6324e\") "
Oct 30 22:21:17.321665 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:17.321634    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/147f4f98-ac6a-4b58-8750-a8410dd6324e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/147f4f98-ac6a-4b58-8750-a8410dd6324e-istio-envoy\") pod \"147f4f98-ac6a-4b58-8750-a8410dd6324e\" (UID: \"147f4f98-ac6a-4b58-8750-a8410dd6324e\") "
Oct 30 22:21:17.321665 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:17.321659    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/147f4f98-ac6a-4b58-8750-a8410dd6324e-kube-api-access-zblzp\" (UniqueName: \"kubernetes.io/projected/147f4f98-ac6a-4b58-8750-a8410dd6324e-kube-api-access-zblzp\") pod \"147f4f98-ac6a-4b58-8750-a8410dd6324e\" (UID: \"147f4f98-ac6a-4b58-8750-a8410dd6324e\") "
Oct 30 22:21:17.321906 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:17.321685    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/147f4f98-ac6a-4b58-8750-a8410dd6324e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/147f4f98-ac6a-4b58-8750-a8410dd6324e-istio-podinfo\") pod \"147f4f98-ac6a-4b58-8750-a8410dd6324e\" (UID: \"147f4f98-ac6a-4b58-8750-a8410dd6324e\") "
Oct 30 22:21:17.321906 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:17.321743    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/147f4f98-ac6a-4b58-8750-a8410dd6324e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/147f4f98-ac6a-4b58-8750-a8410dd6324e-workload-socket\") pod \"147f4f98-ac6a-4b58-8750-a8410dd6324e\" (UID: \"147f4f98-ac6a-4b58-8750-a8410dd6324e\") "
Oct 30 22:21:17.321906 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:17.321775    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/147f4f98-ac6a-4b58-8750-a8410dd6324e-istio-token\" (UniqueName: \"kubernetes.io/projected/147f4f98-ac6a-4b58-8750-a8410dd6324e-istio-token\") pod \"147f4f98-ac6a-4b58-8750-a8410dd6324e\" (UID: \"147f4f98-ac6a-4b58-8750-a8410dd6324e\") "
Oct 30 22:21:17.322069 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:17.321770    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/147f4f98-ac6a-4b58-8750-a8410dd6324e-workload-certs" pod "147f4f98-ac6a-4b58-8750-a8410dd6324e" (UID: "147f4f98-ac6a-4b58-8750-a8410dd6324e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:21:17.322069 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:17.321803    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/147f4f98-ac6a-4b58-8750-a8410dd6324e-istio-data" pod "147f4f98-ac6a-4b58-8750-a8410dd6324e" (UID: "147f4f98-ac6a-4b58-8750-a8410dd6324e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:21:17.322069 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:17.322001    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/147f4f98-ac6a-4b58-8750-a8410dd6324e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/147f4f98-ac6a-4b58-8750-a8410dd6324e-istiod-ca-cert\") pod \"147f4f98-ac6a-4b58-8750-a8410dd6324e\" (UID: \"147f4f98-ac6a-4b58-8750-a8410dd6324e\") "
Oct 30 22:21:17.322069 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:17.322005    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/147f4f98-ac6a-4b58-8750-a8410dd6324e-credential-socket" pod "147f4f98-ac6a-4b58-8750-a8410dd6324e" (UID: "147f4f98-ac6a-4b58-8750-a8410dd6324e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:21:17.322292 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:17.322109    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/147f4f98-ac6a-4b58-8750-a8410dd6324e-workload-socket" pod "147f4f98-ac6a-4b58-8750-a8410dd6324e" (UID: "147f4f98-ac6a-4b58-8750-a8410dd6324e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:21:17.322466 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:17.322444    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/147f4f98-ac6a-4b58-8750-a8410dd6324e-istiod-ca-cert" pod "147f4f98-ac6a-4b58-8750-a8410dd6324e" (UID: "147f4f98-ac6a-4b58-8750-a8410dd6324e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:21:17.322678 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:17.322657    5061 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/147f4f98-ac6a-4b58-8750-a8410dd6324e-workload-certs\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:21:17.322759 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:17.322683    5061 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/147f4f98-ac6a-4b58-8750-a8410dd6324e-istio-data\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:21:17.322759 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:17.322700    5061 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/147f4f98-ac6a-4b58-8750-a8410dd6324e-credential-socket\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:21:17.322759 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:17.322714    5061 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/147f4f98-ac6a-4b58-8750-a8410dd6324e-workload-socket\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:21:17.322759 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:17.322729    5061 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/147f4f98-ac6a-4b58-8750-a8410dd6324e-istiod-ca-cert\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:21:17.324040 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:17.324014    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/147f4f98-ac6a-4b58-8750-a8410dd6324e-istio-podinfo" pod "147f4f98-ac6a-4b58-8750-a8410dd6324e" (UID: "147f4f98-ac6a-4b58-8750-a8410dd6324e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:21:17.324154 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:17.324072    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/147f4f98-ac6a-4b58-8750-a8410dd6324e-kube-api-access-zblzp" pod "147f4f98-ac6a-4b58-8750-a8410dd6324e" (UID: "147f4f98-ac6a-4b58-8750-a8410dd6324e"). InnerVolumeSpecName "kube-api-access-zblzp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:21:17.324154 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:17.324113    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/147f4f98-ac6a-4b58-8750-a8410dd6324e-istio-token" pod "147f4f98-ac6a-4b58-8750-a8410dd6324e" (UID: "147f4f98-ac6a-4b58-8750-a8410dd6324e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:21:17.324154 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:17.324113    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/147f4f98-ac6a-4b58-8750-a8410dd6324e-istio-envoy" pod "147f4f98-ac6a-4b58-8750-a8410dd6324e" (UID: "147f4f98-ac6a-4b58-8750-a8410dd6324e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:21:17.423690 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:17.423663    5061 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/147f4f98-ac6a-4b58-8750-a8410dd6324e-istio-envoy\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:21:17.423690 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:17.423689    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zblzp\" (UniqueName: \"kubernetes.io/projected/147f4f98-ac6a-4b58-8750-a8410dd6324e-kube-api-access-zblzp\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:21:17.423880 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:17.423699    5061 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/147f4f98-ac6a-4b58-8750-a8410dd6324e-istio-podinfo\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:21:17.423880 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:17.423708    5061 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/147f4f98-ac6a-4b58-8750-a8410dd6324e-istio-token\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:21:18.115355 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:18.115322    5061 generic.go:356] "Generic (PLEG): container finished" podID="147f4f98-ac6a-4b58-8750-a8410dd6324e" containerID="ec9430b1c8a6e0956573be159466e09daa706f3ceba7cafd80697c5747f4b8fc" exitCode=0
Oct 30 22:21:18.115520 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:18.115379    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-chgcq"
Oct 30 22:21:18.115520 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:18.115399    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-chgcq" event={"ID":"147f4f98-ac6a-4b58-8750-a8410dd6324e","Type":"ContainerDied","Data":"ec9430b1c8a6e0956573be159466e09daa706f3ceba7cafd80697c5747f4b8fc"}
Oct 30 22:21:18.115520 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:18.115457    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-chgcq" event={"ID":"147f4f98-ac6a-4b58-8750-a8410dd6324e","Type":"ContainerDied","Data":"da02c9b461cfd4e81fb785512caf1f4ca215538b8615a90c6f288e516075fd27"}
Oct 30 22:21:18.115520 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:18.115480    5061 scope.go:122] "RemoveContainer" containerID="ec9430b1c8a6e0956573be159466e09daa706f3ceba7cafd80697c5747f4b8fc"
Oct 30 22:21:18.132868 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:18.132845    5061 scope.go:122] "RemoveContainer" containerID="0691492afd2aebc1e5aa04014c2805dfd67ff4e3087e09d2d5a432f20c862d05"
Oct 30 22:21:18.142199 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:18.142180    5061 scope.go:122] "RemoveContainer" containerID="ec9430b1c8a6e0956573be159466e09daa706f3ceba7cafd80697c5747f4b8fc"
Oct 30 22:21:18.142475 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:21:18.142452    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec9430b1c8a6e0956573be159466e09daa706f3ceba7cafd80697c5747f4b8fc\": container with ID starting with ec9430b1c8a6e0956573be159466e09daa706f3ceba7cafd80697c5747f4b8fc not found: ID does not exist" containerID="ec9430b1c8a6e0956573be159466e09daa706f3ceba7cafd80697c5747f4b8fc"
Oct 30 22:21:18.142566 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:18.142486    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec9430b1c8a6e0956573be159466e09daa706f3ceba7cafd80697c5747f4b8fc"} err="failed to get container status \"ec9430b1c8a6e0956573be159466e09daa706f3ceba7cafd80697c5747f4b8fc\": rpc error: code = NotFound desc = could not find container \"ec9430b1c8a6e0956573be159466e09daa706f3ceba7cafd80697c5747f4b8fc\": container with ID starting with ec9430b1c8a6e0956573be159466e09daa706f3ceba7cafd80697c5747f4b8fc not found: ID does not exist"
Oct 30 22:21:18.142566 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:18.142512    5061 scope.go:122] "RemoveContainer" containerID="0691492afd2aebc1e5aa04014c2805dfd67ff4e3087e09d2d5a432f20c862d05"
Oct 30 22:21:18.142886 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:21:18.142867    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0691492afd2aebc1e5aa04014c2805dfd67ff4e3087e09d2d5a432f20c862d05\": container with ID starting with 0691492afd2aebc1e5aa04014c2805dfd67ff4e3087e09d2d5a432f20c862d05 not found: ID does not exist" containerID="0691492afd2aebc1e5aa04014c2805dfd67ff4e3087e09d2d5a432f20c862d05"
Oct 30 22:21:18.142932 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:18.142893    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0691492afd2aebc1e5aa04014c2805dfd67ff4e3087e09d2d5a432f20c862d05"} err="failed to get container status \"0691492afd2aebc1e5aa04014c2805dfd67ff4e3087e09d2d5a432f20c862d05\": rpc error: code = NotFound desc = could not find container \"0691492afd2aebc1e5aa04014c2805dfd67ff4e3087e09d2d5a432f20c862d05\": container with ID starting with 0691492afd2aebc1e5aa04014c2805dfd67ff4e3087e09d2d5a432f20c862d05 not found: ID does not exist"
Oct 30 22:21:18.279798 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:18.278593    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-chgcq"]
Oct 30 22:21:18.323778 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:18.323743    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-chgcq"]
Oct 30 22:21:18.530038 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:18.529957    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="147f4f98-ac6a-4b58-8750-a8410dd6324e" path="/var/lib/kubelet/pods/147f4f98-ac6a-4b58-8750-a8410dd6324e/volumes"
Oct 30 22:21:24.667744 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:21:24.667705    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 22:21:25.093618 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:25.093588    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-84a3cf-6c95488bd4-cjmzh"
Oct 30 22:21:25.093802 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:25.093727    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-84a3cf-6cb65fb6cf-r64nq"
Oct 30 22:21:28.682280 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:28.682247    5061 scope.go:122] "RemoveContainer" containerID="9db031599672d729bea66c018b477c94a91be9a60623547868bb62fc1213d37a"
Oct 30 22:21:30.408880 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:30.408848    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-e66edb-jnj2r"]
Oct 30 22:21:30.409597 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:30.409579    5061 cpu_manager.go:414] "containerMap: removing container" podUID="147f4f98-ac6a-4b58-8750-a8410dd6324e" containerName="istio-proxy"
Oct 30 22:21:30.409597 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:30.409597    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="147f4f98-ac6a-4b58-8750-a8410dd6324e" containerName="istio-proxy"
Oct 30 22:21:30.409710 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:30.409648    5061 cpu_manager.go:414] "containerMap: removing container" podUID="147f4f98-ac6a-4b58-8750-a8410dd6324e" containerName="istio-proxy"
Oct 30 22:21:30.409710 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:30.409653    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="147f4f98-ac6a-4b58-8750-a8410dd6324e" containerName="istio-proxy"
Oct 30 22:21:30.409776 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:30.409745    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="147f4f98-ac6a-4b58-8750-a8410dd6324e" containerName="istio-proxy"
Oct 30 22:21:30.409776 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:30.409760    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="147f4f98-ac6a-4b58-8750-a8410dd6324e" containerName="istio-proxy"
Oct 30 22:21:30.414999 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:30.414980    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-e66edb-jnj2r"
Oct 30 22:21:30.465685 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:30.465661    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:21:30.469807 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:30.469780    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-e66edb-jnj2r"]
Oct 30 22:21:30.502122 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:30.502092    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q652c\" (UniqueName: \"kubernetes.io/projected/8d1c4a44-dc12-46ea-92d8-07ef25a171f1-kube-api-access-q652c\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-e66edb-jnj2r\" (UID: \"8d1c4a44-dc12-46ea-92d8-07ef25a171f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-e66edb-jnj2r"
Oct 30 22:21:30.502274 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:30.502251    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d1c4a44-dc12-46ea-92d8-07ef25a171f1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-e66edb-jnj2r\" (UID: \"8d1c4a44-dc12-46ea-92d8-07ef25a171f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-e66edb-jnj2r"
Oct 30 22:21:30.502408 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:30.502385    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d1c4a44-dc12-46ea-92d8-07ef25a171f1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-e66edb-jnj2r\" (UID: \"8d1c4a44-dc12-46ea-92d8-07ef25a171f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-e66edb-jnj2r"
Oct 30 22:21:30.512624 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:30.512599    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gdmfg\""
Oct 30 22:21:30.603939 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:30.603891    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d1c4a44-dc12-46ea-92d8-07ef25a171f1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-e66edb-jnj2r\" (UID: \"8d1c4a44-dc12-46ea-92d8-07ef25a171f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-e66edb-jnj2r"
Oct 30 22:21:30.604112 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:30.603986    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d1c4a44-dc12-46ea-92d8-07ef25a171f1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-e66edb-jnj2r\" (UID: \"8d1c4a44-dc12-46ea-92d8-07ef25a171f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-e66edb-jnj2r"
Oct 30 22:21:30.604188 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:30.604156    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q652c\" (UniqueName: \"kubernetes.io/projected/8d1c4a44-dc12-46ea-92d8-07ef25a171f1-kube-api-access-q652c\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-e66edb-jnj2r\" (UID: \"8d1c4a44-dc12-46ea-92d8-07ef25a171f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-e66edb-jnj2r"
Oct 30 22:21:30.604672 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:30.604645    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d1c4a44-dc12-46ea-92d8-07ef25a171f1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-e66edb-jnj2r\" (UID: \"8d1c4a44-dc12-46ea-92d8-07ef25a171f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-e66edb-jnj2r"
Oct 30 22:21:30.606387 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:30.606363    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d1c4a44-dc12-46ea-92d8-07ef25a171f1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-e66edb-jnj2r\" (UID: \"8d1c4a44-dc12-46ea-92d8-07ef25a171f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-e66edb-jnj2r"
Oct 30 22:21:30.659031 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:30.658975    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q652c\" (UniqueName: \"kubernetes.io/projected/8d1c4a44-dc12-46ea-92d8-07ef25a171f1-kube-api-access-q652c\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-e66edb-jnj2r\" (UID: \"8d1c4a44-dc12-46ea-92d8-07ef25a171f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-e66edb-jnj2r"
Oct 30 22:21:30.728696 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:30.728668    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-e66edb-jnj2r"
Oct 30 22:21:31.097419 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:31.097390    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-e66edb-jnj2r"]
Oct 30 22:21:31.098017 ip-10-0-130-60 kubenswrapper[5061]: W1030 22:21:31.097978    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d1c4a44_dc12_46ea_92d8_07ef25a171f1.slice/crio-c3d23a3bd27015142ef573a37dd0ba5cf507e35187d971a131245a3c1d488184 WatchSource:0}: Error finding container c3d23a3bd27015142ef573a37dd0ba5cf507e35187d971a131245a3c1d488184: Status 404 returned error can't find the container with id c3d23a3bd27015142ef573a37dd0ba5cf507e35187d971a131245a3c1d488184
Oct 30 22:21:31.215012 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:31.214975    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-e66edb-jnj2r" event={"ID":"8d1c4a44-dc12-46ea-92d8-07ef25a171f1","Type":"ContainerStarted","Data":"c3d23a3bd27015142ef573a37dd0ba5cf507e35187d971a131245a3c1d488184"}
Oct 30 22:21:32.228940 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:32.228905    5061 generic.go:356] "Generic (PLEG): container finished" podID="8d1c4a44-dc12-46ea-92d8-07ef25a171f1" containerID="f9029a35a96288f6f645c0720c4912faf866e95e51535c6c1cb5fb3c0834a121" exitCode=0
Oct 30 22:21:32.229343 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:32.228990    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-e66edb-jnj2r" event={"ID":"8d1c4a44-dc12-46ea-92d8-07ef25a171f1","Type":"ContainerDied","Data":"f9029a35a96288f6f645c0720c4912faf866e95e51535c6c1cb5fb3c0834a121"}
Oct 30 22:21:33.390779 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:33.390754    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-e66edb-jnj2r"
Oct 30 22:21:33.541948 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:33.541853    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d1c4a44-dc12-46ea-92d8-07ef25a171f1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d1c4a44-dc12-46ea-92d8-07ef25a171f1-maas-api-xks-ca\") pod \"8d1c4a44-dc12-46ea-92d8-07ef25a171f1\" (UID: \"8d1c4a44-dc12-46ea-92d8-07ef25a171f1\") "
Oct 30 22:21:33.542121 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:33.542028    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d1c4a44-dc12-46ea-92d8-07ef25a171f1-kube-api-access-q652c\" (UniqueName: \"kubernetes.io/projected/8d1c4a44-dc12-46ea-92d8-07ef25a171f1-kube-api-access-q652c\") pod \"8d1c4a44-dc12-46ea-92d8-07ef25a171f1\" (UID: \"8d1c4a44-dc12-46ea-92d8-07ef25a171f1\") "
Oct 30 22:21:33.542199 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:33.542185    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d1c4a44-dc12-46ea-92d8-07ef25a171f1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d1c4a44-dc12-46ea-92d8-07ef25a171f1-maas-api-openshift-ca\") pod \"8d1c4a44-dc12-46ea-92d8-07ef25a171f1\" (UID: \"8d1c4a44-dc12-46ea-92d8-07ef25a171f1\") "
Oct 30 22:21:33.542715 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:33.542620    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d1c4a44-dc12-46ea-92d8-07ef25a171f1-maas-api-openshift-ca" pod "8d1c4a44-dc12-46ea-92d8-07ef25a171f1" (UID: "8d1c4a44-dc12-46ea-92d8-07ef25a171f1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:21:33.543045 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:33.543021    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d1c4a44-dc12-46ea-92d8-07ef25a171f1-maas-api-openshift-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:21:33.544096 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:33.544064    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d1c4a44-dc12-46ea-92d8-07ef25a171f1-maas-api-xks-ca" pod "8d1c4a44-dc12-46ea-92d8-07ef25a171f1" (UID: "8d1c4a44-dc12-46ea-92d8-07ef25a171f1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:21:33.544583 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:33.544559    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d1c4a44-dc12-46ea-92d8-07ef25a171f1-kube-api-access-q652c" pod "8d1c4a44-dc12-46ea-92d8-07ef25a171f1" (UID: "8d1c4a44-dc12-46ea-92d8-07ef25a171f1"). InnerVolumeSpecName "kube-api-access-q652c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:21:33.644231 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:33.644201    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d1c4a44-dc12-46ea-92d8-07ef25a171f1-maas-api-xks-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:21:33.644231 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:33.644230    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q652c\" (UniqueName: \"kubernetes.io/projected/8d1c4a44-dc12-46ea-92d8-07ef25a171f1-kube-api-access-q652c\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:21:34.244314 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:34.244279    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-e66edb-jnj2r"
Oct 30 22:21:34.244487 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:34.244332    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-e66edb-jnj2r" event={"ID":"8d1c4a44-dc12-46ea-92d8-07ef25a171f1","Type":"ContainerDied","Data":"c3d23a3bd27015142ef573a37dd0ba5cf507e35187d971a131245a3c1d488184"}
Oct 30 22:21:34.244487 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:34.244378    5061 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c3d23a3bd27015142ef573a37dd0ba5cf507e35187d971a131245a3c1d488184"
Oct 30 22:21:42.054681 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:42.054643    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f24db166-hnbss"]
Oct 30 22:21:42.102346 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:42.102305    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f24db166-hnbss"]
Oct 30 22:21:42.529828 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:42.529744    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d2a7ba6-ea4c-4425-88c2-415faa0b0db5" path="/var/lib/kubelet/pods/7d2a7ba6-ea4c-4425-88c2-415faa0b0db5/volumes"
Oct 30 22:21:48.609638 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:48.609603    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-778d39d3-4txvg"]
Oct 30 22:21:48.656345 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:48.656311    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-778d39d3-4txvg"]
Oct 30 22:21:50.530238 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:21:50.530199    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="339da026-9f6c-48d0-8d77-6f2fd7675c75" path="/var/lib/kubelet/pods/339da026-9f6c-48d0-8d77-6f2fd7675c75/volumes"
Oct 30 22:21:54.668686 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:21:54.668648    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 22:22:08.563215 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:08.563139    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c1e2ab-5b47865787-wj8g9"]
Oct 30 22:22:08.563796 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:08.563781    5061 cpu_manager.go:414] "containerMap: removing container" podUID="8d1c4a44-dc12-46ea-92d8-07ef25a171f1" containerName="revoke-keys"
Oct 30 22:22:08.563796 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:08.563796    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d1c4a44-dc12-46ea-92d8-07ef25a171f1" containerName="revoke-keys"
Oct 30 22:22:08.563895 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:08.563885    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d1c4a44-dc12-46ea-92d8-07ef25a171f1" containerName="revoke-keys"
Oct 30 22:22:08.567841 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:08.567819    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c1e2ab-5b47865787-wj8g9"
Oct 30 22:22:08.614846 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:08.614810    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/136b259c-03a5-4cc9-8702-fd78a552bb65-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-c1e2ab-5b47865787-wj8g9\" (UID: \"136b259c-03a5-4cc9-8702-fd78a552bb65\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c1e2ab-5b47865787-wj8g9"
Oct 30 22:22:08.615005 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:08.614911    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpjj7\" (UniqueName: \"kubernetes.io/projected/136b259c-03a5-4cc9-8702-fd78a552bb65-kube-api-access-dpjj7\") pod \"payload-pre-processing-e2e-api-b-c1e2ab-5b47865787-wj8g9\" (UID: \"136b259c-03a5-4cc9-8702-fd78a552bb65\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c1e2ab-5b47865787-wj8g9"
Oct 30 22:22:08.619480 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:08.619455    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-c1e2ab\""
Oct 30 22:22:08.683651 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:08.676857    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-c1e2ab-dockercfg-fwtvx\""
Oct 30 22:22:08.693142 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:08.693104    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c1e2ab-5b47865787-wj8g9"]
Oct 30 22:22:08.728630 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:08.722713    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/136b259c-03a5-4cc9-8702-fd78a552bb65-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-c1e2ab-5b47865787-wj8g9\" (UID: \"136b259c-03a5-4cc9-8702-fd78a552bb65\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c1e2ab-5b47865787-wj8g9"
Oct 30 22:22:08.728630 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:08.722838    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpjj7\" (UniqueName: \"kubernetes.io/projected/136b259c-03a5-4cc9-8702-fd78a552bb65-kube-api-access-dpjj7\") pod \"payload-pre-processing-e2e-api-b-c1e2ab-5b47865787-wj8g9\" (UID: \"136b259c-03a5-4cc9-8702-fd78a552bb65\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c1e2ab-5b47865787-wj8g9"
Oct 30 22:22:08.728630 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:08.727079    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/136b259c-03a5-4cc9-8702-fd78a552bb65-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-c1e2ab-5b47865787-wj8g9\" (UID: \"136b259c-03a5-4cc9-8702-fd78a552bb65\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c1e2ab-5b47865787-wj8g9"
Oct 30 22:22:08.783965 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:08.783932    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpjj7\" (UniqueName: \"kubernetes.io/projected/136b259c-03a5-4cc9-8702-fd78a552bb65-kube-api-access-dpjj7\") pod \"payload-pre-processing-e2e-api-b-c1e2ab-5b47865787-wj8g9\" (UID: \"136b259c-03a5-4cc9-8702-fd78a552bb65\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c1e2ab-5b47865787-wj8g9"
Oct 30 22:22:08.881685 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:08.881571    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c1e2ab-5b47865787-wj8g9"
Oct 30 22:22:09.258497 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:09.258469    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c1e2ab-5b47865787-wj8g9"]
Oct 30 22:22:09.261258 ip-10-0-130-60 kubenswrapper[5061]: W1030 22:22:09.261233    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod136b259c_03a5_4cc9_8702_fd78a552bb65.slice/crio-b8c734a100f5ac7f93d999869b15c0a78cee8ed886b279a6af6c8a2cd76795f2 WatchSource:0}: Error finding container b8c734a100f5ac7f93d999869b15c0a78cee8ed886b279a6af6c8a2cd76795f2: Status 404 returned error can't find the container with id b8c734a100f5ac7f93d999869b15c0a78cee8ed886b279a6af6c8a2cd76795f2
Oct 30 22:22:09.560127 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:09.560084    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c1e2ab-5b47865787-wj8g9" event={"ID":"136b259c-03a5-4cc9-8702-fd78a552bb65","Type":"ContainerStarted","Data":"e4993a4a9426eaf27fda7cf07b650517de53f9947aef942d328a84b6bf9faa89"}
Oct 30 22:22:09.560334 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:09.560146    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c1e2ab-5b47865787-wj8g9" event={"ID":"136b259c-03a5-4cc9-8702-fd78a552bb65","Type":"ContainerStarted","Data":"b8c734a100f5ac7f93d999869b15c0a78cee8ed886b279a6af6c8a2cd76795f2"}
Oct 30 22:22:09.560334 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:09.560178    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c1e2ab-5b47865787-wj8g9"
Oct 30 22:22:09.675271 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:09.675218    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c1e2ab-5b47865787-wj8g9" podStartSLOduration=1.67520429 podStartE2EDuration="1.67520429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:22:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:22:09.671768365 +0000 UTC m=+2383.284538122" watchObservedRunningTime="2026-10-30 22:22:09.67520429 +0000 UTC m=+2383.287974079"
Oct 30 22:22:20.571336 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:20.571307    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c1e2ab-5b47865787-wj8g9"
Oct 30 22:22:24.667845 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:22:24.667808    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 22:22:26.883841 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:26.883814    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-sclrt_5e74071e-0462-42b5-a05c-d90bb638478a/console-operator/2.log"
Oct 30 22:22:26.915060 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:26.915036    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-sclrt_5e74071e-0462-42b5-a05c-d90bb638478a/console-operator/2.log"
Oct 30 22:22:28.732543 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:28.732499    5061 scope.go:122] "RemoveContainer" containerID="5571afc0fad36f78a34143e095c058d4b26ec985bdbce5b8136a7e81f50c0f52"
Oct 30 22:22:28.742888 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:28.742866    5061 scope.go:122] "RemoveContainer" containerID="1c31084efb8058bc104bc1c2190660c8bba6910951b0afba1a9792d0cabe57ad"
Oct 30 22:22:28.753452 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:28.753420    5061 scope.go:122] "RemoveContainer" containerID="1655dcb183590773dc6a0807ee1b18d3088151835562d856752f3d4644208e06"
Oct 30 22:22:31.964988 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:31.964945    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a07620-7m6w7"]
Oct 30 22:22:31.971288 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:31.971267    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a07620-7m6w7"
Oct 30 22:22:31.975554 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:31.975513    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/848948cb-8b8b-4dc7-891d-14b18e9143d8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-a07620-7m6w7\" (UID: \"848948cb-8b8b-4dc7-891d-14b18e9143d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a07620-7m6w7"
Oct 30 22:22:31.975684 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:31.975612    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ss4pz\" (UniqueName: \"kubernetes.io/projected/848948cb-8b8b-4dc7-891d-14b18e9143d8-kube-api-access-ss4pz\") pod \"maas-api-revoke-keys-e2e-worker-w1-a07620-7m6w7\" (UID: \"848948cb-8b8b-4dc7-891d-14b18e9143d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a07620-7m6w7"
Oct 30 22:22:31.975684 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:31.975674    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/848948cb-8b8b-4dc7-891d-14b18e9143d8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-a07620-7m6w7\" (UID: \"848948cb-8b8b-4dc7-891d-14b18e9143d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a07620-7m6w7"
Oct 30 22:22:32.021701 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:32.021673    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:22:32.067828 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:32.067802    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gdmfg\""
Oct 30 22:22:32.073722 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:32.073699    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a07620-7m6w7"]
Oct 30 22:22:32.076606 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:32.076585    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/848948cb-8b8b-4dc7-891d-14b18e9143d8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-a07620-7m6w7\" (UID: \"848948cb-8b8b-4dc7-891d-14b18e9143d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a07620-7m6w7"
Oct 30 22:22:32.076729 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:32.076632    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ss4pz\" (UniqueName: \"kubernetes.io/projected/848948cb-8b8b-4dc7-891d-14b18e9143d8-kube-api-access-ss4pz\") pod \"maas-api-revoke-keys-e2e-worker-w1-a07620-7m6w7\" (UID: \"848948cb-8b8b-4dc7-891d-14b18e9143d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a07620-7m6w7"
Oct 30 22:22:32.076729 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:32.076693    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/848948cb-8b8b-4dc7-891d-14b18e9143d8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-a07620-7m6w7\" (UID: \"848948cb-8b8b-4dc7-891d-14b18e9143d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a07620-7m6w7"
Oct 30 22:22:32.077435 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:32.077409    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/848948cb-8b8b-4dc7-891d-14b18e9143d8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-a07620-7m6w7\" (UID: \"848948cb-8b8b-4dc7-891d-14b18e9143d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a07620-7m6w7"
Oct 30 22:22:32.079205 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:32.079170    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/848948cb-8b8b-4dc7-891d-14b18e9143d8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-a07620-7m6w7\" (UID: \"848948cb-8b8b-4dc7-891d-14b18e9143d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a07620-7m6w7"
Oct 30 22:22:32.131714 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:32.131682    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ss4pz\" (UniqueName: \"kubernetes.io/projected/848948cb-8b8b-4dc7-891d-14b18e9143d8-kube-api-access-ss4pz\") pod \"maas-api-revoke-keys-e2e-worker-w1-a07620-7m6w7\" (UID: \"848948cb-8b8b-4dc7-891d-14b18e9143d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a07620-7m6w7"
Oct 30 22:22:32.284770 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:32.284677    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a07620-7m6w7"
Oct 30 22:22:32.656601 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:32.656391    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a07620-7m6w7"]
Oct 30 22:22:32.657748 ip-10-0-130-60 kubenswrapper[5061]: W1030 22:22:32.657720    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod848948cb_8b8b_4dc7_891d_14b18e9143d8.slice/crio-12d9fa79a27cd99b279dca6242900d071becbc87372939e0475d0a3e1ebef349 WatchSource:0}: Error finding container 12d9fa79a27cd99b279dca6242900d071becbc87372939e0475d0a3e1ebef349: Status 404 returned error can't find the container with id 12d9fa79a27cd99b279dca6242900d071becbc87372939e0475d0a3e1ebef349
Oct 30 22:22:32.783491 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:32.783462    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a07620-7m6w7" event={"ID":"848948cb-8b8b-4dc7-891d-14b18e9143d8","Type":"ContainerStarted","Data":"12d9fa79a27cd99b279dca6242900d071becbc87372939e0475d0a3e1ebef349"}
Oct 30 22:22:33.791728 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:33.791696    5061 generic.go:356] "Generic (PLEG): container finished" podID="848948cb-8b8b-4dc7-891d-14b18e9143d8" containerID="96619c69d6306a1554b4d5356b658d3d6ed92c4a7cd756fdbc41720fc21eea5f" exitCode=0
Oct 30 22:22:33.792142 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:33.791789    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a07620-7m6w7" event={"ID":"848948cb-8b8b-4dc7-891d-14b18e9143d8","Type":"ContainerDied","Data":"96619c69d6306a1554b4d5356b658d3d6ed92c4a7cd756fdbc41720fc21eea5f"}
Oct 30 22:22:34.974561 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:34.974518    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a07620-7m6w7"
Oct 30 22:22:35.118591 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:35.118493    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/848948cb-8b8b-4dc7-891d-14b18e9143d8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/848948cb-8b8b-4dc7-891d-14b18e9143d8-maas-api-xks-ca\") pod \"848948cb-8b8b-4dc7-891d-14b18e9143d8\" (UID: \"848948cb-8b8b-4dc7-891d-14b18e9143d8\") "
Oct 30 22:22:35.118767 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:35.118596    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/848948cb-8b8b-4dc7-891d-14b18e9143d8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/848948cb-8b8b-4dc7-891d-14b18e9143d8-maas-api-openshift-ca\") pod \"848948cb-8b8b-4dc7-891d-14b18e9143d8\" (UID: \"848948cb-8b8b-4dc7-891d-14b18e9143d8\") "
Oct 30 22:22:35.118850 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:35.118828    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/848948cb-8b8b-4dc7-891d-14b18e9143d8-kube-api-access-ss4pz\" (UniqueName: \"kubernetes.io/projected/848948cb-8b8b-4dc7-891d-14b18e9143d8-kube-api-access-ss4pz\") pod \"848948cb-8b8b-4dc7-891d-14b18e9143d8\" (UID: \"848948cb-8b8b-4dc7-891d-14b18e9143d8\") "
Oct 30 22:22:35.119099 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:35.119075    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/848948cb-8b8b-4dc7-891d-14b18e9143d8-maas-api-openshift-ca" pod "848948cb-8b8b-4dc7-891d-14b18e9143d8" (UID: "848948cb-8b8b-4dc7-891d-14b18e9143d8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:22:35.119489 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:35.119460    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/848948cb-8b8b-4dc7-891d-14b18e9143d8-maas-api-openshift-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:35.121282 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:35.121248    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/848948cb-8b8b-4dc7-891d-14b18e9143d8-kube-api-access-ss4pz" pod "848948cb-8b8b-4dc7-891d-14b18e9143d8" (UID: "848948cb-8b8b-4dc7-891d-14b18e9143d8"). InnerVolumeSpecName "kube-api-access-ss4pz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:22:35.121394 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:35.121291    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/848948cb-8b8b-4dc7-891d-14b18e9143d8-maas-api-xks-ca" pod "848948cb-8b8b-4dc7-891d-14b18e9143d8" (UID: "848948cb-8b8b-4dc7-891d-14b18e9143d8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:22:35.192518 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:35.192487    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-e66edb-jnj2r"]
Oct 30 22:22:35.196078 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:35.196050    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-e66edb-jnj2r"]
Oct 30 22:22:35.220800 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:35.220768    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ss4pz\" (UniqueName: \"kubernetes.io/projected/848948cb-8b8b-4dc7-891d-14b18e9143d8-kube-api-access-ss4pz\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:35.220916 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:35.220804    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/848948cb-8b8b-4dc7-891d-14b18e9143d8-maas-api-xks-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:35.807653 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:35.807612    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a07620-7m6w7" event={"ID":"848948cb-8b8b-4dc7-891d-14b18e9143d8","Type":"ContainerDied","Data":"12d9fa79a27cd99b279dca6242900d071becbc87372939e0475d0a3e1ebef349"}
Oct 30 22:22:35.807653 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:35.807639    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a07620-7m6w7"
Oct 30 22:22:35.807866 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:35.807703    5061 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="12d9fa79a27cd99b279dca6242900d071becbc87372939e0475d0a3e1ebef349"
Oct 30 22:22:35.817358 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:22:35.817333    5061 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod848948cb_8b8b_4dc7_891d_14b18e9143d8.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 22:22:36.369956 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:36.369920    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-a07620/simulator-w1-a07620-kserve-55b44674f4-4z9lk"]
Oct 30 22:22:36.530756 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:36.530724    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d1c4a44-dc12-46ea-92d8-07ef25a171f1" path="/var/lib/kubelet/pods/8d1c4a44-dc12-46ea-92d8-07ef25a171f1/volumes"
Oct 30 22:22:42.963148 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:42.963112    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-5be0d1-jzdjw"]
Oct 30 22:22:42.964071 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:42.964053    5061 cpu_manager.go:414] "containerMap: removing container" podUID="848948cb-8b8b-4dc7-891d-14b18e9143d8" containerName="revoke-keys"
Oct 30 22:22:42.964151 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:42.964073    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="848948cb-8b8b-4dc7-891d-14b18e9143d8" containerName="revoke-keys"
Oct 30 22:22:42.964231 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:42.964220    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="848948cb-8b8b-4dc7-891d-14b18e9143d8" containerName="revoke-keys"
Oct 30 22:22:42.969460 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:42.969437    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-5be0d1-jzdjw"
Oct 30 22:22:43.022287 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:43.022258    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:22:43.047270 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:43.047236    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17eb78ae-158c-49f2-8ff2-b7f1a7918a82-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-5be0d1-jzdjw\" (UID: \"17eb78ae-158c-49f2-8ff2-b7f1a7918a82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-5be0d1-jzdjw"
Oct 30 22:22:43.047426 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:43.047356    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17eb78ae-158c-49f2-8ff2-b7f1a7918a82-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-5be0d1-jzdjw\" (UID: \"17eb78ae-158c-49f2-8ff2-b7f1a7918a82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-5be0d1-jzdjw"
Oct 30 22:22:43.047670 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:43.047642    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ffpk\" (UniqueName: \"kubernetes.io/projected/17eb78ae-158c-49f2-8ff2-b7f1a7918a82-kube-api-access-8ffpk\") pod \"maas-api-revoke-keys-e2e-worker-w2-5be0d1-jzdjw\" (UID: \"17eb78ae-158c-49f2-8ff2-b7f1a7918a82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-5be0d1-jzdjw"
Oct 30 22:22:43.066785 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:43.066753    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gdmfg\""
Oct 30 22:22:43.073094 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:43.073068    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-5be0d1-jzdjw"]
Oct 30 22:22:43.148511 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:43.148477    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ffpk\" (UniqueName: \"kubernetes.io/projected/17eb78ae-158c-49f2-8ff2-b7f1a7918a82-kube-api-access-8ffpk\") pod \"maas-api-revoke-keys-e2e-worker-w2-5be0d1-jzdjw\" (UID: \"17eb78ae-158c-49f2-8ff2-b7f1a7918a82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-5be0d1-jzdjw"
Oct 30 22:22:43.148710 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:43.148599    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17eb78ae-158c-49f2-8ff2-b7f1a7918a82-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-5be0d1-jzdjw\" (UID: \"17eb78ae-158c-49f2-8ff2-b7f1a7918a82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-5be0d1-jzdjw"
Oct 30 22:22:43.148710 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:43.148661    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17eb78ae-158c-49f2-8ff2-b7f1a7918a82-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-5be0d1-jzdjw\" (UID: \"17eb78ae-158c-49f2-8ff2-b7f1a7918a82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-5be0d1-jzdjw"
Oct 30 22:22:43.149379 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:43.149357    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17eb78ae-158c-49f2-8ff2-b7f1a7918a82-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-5be0d1-jzdjw\" (UID: \"17eb78ae-158c-49f2-8ff2-b7f1a7918a82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-5be0d1-jzdjw"
Oct 30 22:22:43.151129 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:43.151107    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17eb78ae-158c-49f2-8ff2-b7f1a7918a82-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-5be0d1-jzdjw\" (UID: \"17eb78ae-158c-49f2-8ff2-b7f1a7918a82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-5be0d1-jzdjw"
Oct 30 22:22:43.204916 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:43.204883    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ffpk\" (UniqueName: \"kubernetes.io/projected/17eb78ae-158c-49f2-8ff2-b7f1a7918a82-kube-api-access-8ffpk\") pod \"maas-api-revoke-keys-e2e-worker-w2-5be0d1-jzdjw\" (UID: \"17eb78ae-158c-49f2-8ff2-b7f1a7918a82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-5be0d1-jzdjw"
Oct 30 22:22:43.283814 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:43.283734    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-5be0d1-jzdjw"
Oct 30 22:22:43.662856 ip-10-0-130-60 kubenswrapper[5061]: W1030 22:22:43.662828    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17eb78ae_158c_49f2_8ff2_b7f1a7918a82.slice/crio-ec0104aa5afee0ee364e6db04d5fd2ce356ad2f6902e7795f6469ee5da5715af WatchSource:0}: Error finding container ec0104aa5afee0ee364e6db04d5fd2ce356ad2f6902e7795f6469ee5da5715af: Status 404 returned error can't find the container with id ec0104aa5afee0ee364e6db04d5fd2ce356ad2f6902e7795f6469ee5da5715af
Oct 30 22:22:43.663752 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:43.663728    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-5be0d1-jzdjw"]
Oct 30 22:22:43.865669 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:43.865632    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-5be0d1-jzdjw" event={"ID":"17eb78ae-158c-49f2-8ff2-b7f1a7918a82","Type":"ContainerStarted","Data":"ec0104aa5afee0ee364e6db04d5fd2ce356ad2f6902e7795f6469ee5da5715af"}
Oct 30 22:22:44.876924 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:44.876886    5061 generic.go:356] "Generic (PLEG): container finished" podID="17eb78ae-158c-49f2-8ff2-b7f1a7918a82" containerID="dce44c1fcb98d392020ac5b5f3e5412e255ed47bef466e5457f8cf4ae0f9f0bc" exitCode=0
Oct 30 22:22:44.877360 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:44.876956    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-5be0d1-jzdjw" event={"ID":"17eb78ae-158c-49f2-8ff2-b7f1a7918a82","Type":"ContainerDied","Data":"dce44c1fcb98d392020ac5b5f3e5412e255ed47bef466e5457f8cf4ae0f9f0bc"}
Oct 30 22:22:46.118100 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:46.118066    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-5be0d1-jzdjw"
Oct 30 22:22:46.199335 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:46.199247    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17eb78ae-158c-49f2-8ff2-b7f1a7918a82-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17eb78ae-158c-49f2-8ff2-b7f1a7918a82-maas-api-xks-ca\") pod \"17eb78ae-158c-49f2-8ff2-b7f1a7918a82\" (UID: \"17eb78ae-158c-49f2-8ff2-b7f1a7918a82\") "
Oct 30 22:22:46.199335 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:46.199305    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17eb78ae-158c-49f2-8ff2-b7f1a7918a82-kube-api-access-8ffpk\" (UniqueName: \"kubernetes.io/projected/17eb78ae-158c-49f2-8ff2-b7f1a7918a82-kube-api-access-8ffpk\") pod \"17eb78ae-158c-49f2-8ff2-b7f1a7918a82\" (UID: \"17eb78ae-158c-49f2-8ff2-b7f1a7918a82\") "
Oct 30 22:22:46.199626 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:46.199493    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17eb78ae-158c-49f2-8ff2-b7f1a7918a82-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17eb78ae-158c-49f2-8ff2-b7f1a7918a82-maas-api-openshift-ca\") pod \"17eb78ae-158c-49f2-8ff2-b7f1a7918a82\" (UID: \"17eb78ae-158c-49f2-8ff2-b7f1a7918a82\") "
Oct 30 22:22:46.199881 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:46.199857    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17eb78ae-158c-49f2-8ff2-b7f1a7918a82-maas-api-openshift-ca" pod "17eb78ae-158c-49f2-8ff2-b7f1a7918a82" (UID: "17eb78ae-158c-49f2-8ff2-b7f1a7918a82"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:22:46.200008 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:46.199991    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17eb78ae-158c-49f2-8ff2-b7f1a7918a82-maas-api-openshift-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:46.201368 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:46.201345    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17eb78ae-158c-49f2-8ff2-b7f1a7918a82-maas-api-xks-ca" pod "17eb78ae-158c-49f2-8ff2-b7f1a7918a82" (UID: "17eb78ae-158c-49f2-8ff2-b7f1a7918a82"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:22:46.201469 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:46.201449    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17eb78ae-158c-49f2-8ff2-b7f1a7918a82-kube-api-access-8ffpk" pod "17eb78ae-158c-49f2-8ff2-b7f1a7918a82" (UID: "17eb78ae-158c-49f2-8ff2-b7f1a7918a82"). InnerVolumeSpecName "kube-api-access-8ffpk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:22:46.301778 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:46.301747    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17eb78ae-158c-49f2-8ff2-b7f1a7918a82-maas-api-xks-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:46.301778 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:46.301776    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ffpk\" (UniqueName: \"kubernetes.io/projected/17eb78ae-158c-49f2-8ff2-b7f1a7918a82-kube-api-access-8ffpk\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:46.892869 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:46.892842    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-5be0d1-jzdjw"
Oct 30 22:22:46.893041 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:46.892840    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-5be0d1-jzdjw" event={"ID":"17eb78ae-158c-49f2-8ff2-b7f1a7918a82","Type":"ContainerDied","Data":"ec0104aa5afee0ee364e6db04d5fd2ce356ad2f6902e7795f6469ee5da5715af"}
Oct 30 22:22:46.893041 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:46.892962    5061 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ec0104aa5afee0ee364e6db04d5fd2ce356ad2f6902e7795f6469ee5da5715af"
Oct 30 22:22:47.412395 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:47.412364    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-5be0d1/distinct-w2-5be0d1-kserve-6b85698c8-sxn9k"]
Oct 30 22:22:47.426343 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:47.426315    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-5be0d1/premium-w2-5be0d1-kserve-759b6fb995-c6gp4"]
Oct 30 22:22:48.413887 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:48.413847    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-9r5c8"]
Oct 30 22:22:48.414278 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:48.414161    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-9r5c8" podUID="83adee7e-3585-4d0d-b1b2-d4b5d3be0588" containerName="istio-proxy" containerID="cri-o://6a6cc21e10bc1e0353ea3b36fe7f807ea2a367e761da2d139b8e0f3b8009c1f0" gracePeriod=30
Oct 30 22:22:48.424941 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:48.424914    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-nv6kl"]
Oct 30 22:22:48.425135 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:48.425114    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-nv6kl" podUID="732b6214-aa8c-45fc-bd06-c0f75a267960" containerName="istio-proxy" containerID="cri-o://9cb0b7e5f37d1b429283bcdb104977b0c36564f7cf0cc9f930c4493a4c93fe50" gracePeriod=30
Oct 30 22:22:48.450977 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:48.450936    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-k5knf"]
Oct 30 22:22:48.451188 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:48.451161    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-k5knf" podUID="3489262b-293e-4dba-8bb2-efa4333ab66d" containerName="istio-proxy" containerID="cri-o://69650f186337a304c285b07f94defc757c4f41455ed19bfc01752e056d037b6c" gracePeriod=30
Oct 30 22:22:48.460698 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:48.460668    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-649qh"]
Oct 30 22:22:48.460915 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:48.460892    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-649qh" podUID="598138a5-2e71-4727-9332-3c604144f0ef" containerName="istio-proxy" containerID="cri-o://b38d0702c46a0f99ecacb9cbe7ed469a5bd18364eb5845fc32519c06825cc329" gracePeriod=30
Oct 30 22:22:51.391592 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:51.391551    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-a07620/simulator-w1-a07620-kserve-55b44674f4-4z9lk" podUID="efe7c063-fb0e-4679-8a7c-13f2a0abaa74" containerName="main" containerID="cri-o://0feba69f7cea4a1d44ba150288760dd74da547313ff80433eec1bf73c2b9c466" gracePeriod=45
Oct 30 22:22:51.736673 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:51.736642    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a07620/simulator-w1-a07620-kserve-55b44674f4-4z9lk"
Oct 30 22:22:51.881235 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:51.881201    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efe7c063-fb0e-4679-8a7c-13f2a0abaa74-home\" (UniqueName: \"kubernetes.io/empty-dir/efe7c063-fb0e-4679-8a7c-13f2a0abaa74-home\") pod \"efe7c063-fb0e-4679-8a7c-13f2a0abaa74\" (UID: \"efe7c063-fb0e-4679-8a7c-13f2a0abaa74\") "
Oct 30 22:22:51.881508 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:51.881472    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efe7c063-fb0e-4679-8a7c-13f2a0abaa74-home" pod "efe7c063-fb0e-4679-8a7c-13f2a0abaa74" (UID: "efe7c063-fb0e-4679-8a7c-13f2a0abaa74"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:22:51.881706 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:51.881665    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efe7c063-fb0e-4679-8a7c-13f2a0abaa74-dshm\" (UniqueName: \"kubernetes.io/empty-dir/efe7c063-fb0e-4679-8a7c-13f2a0abaa74-dshm\") pod \"efe7c063-fb0e-4679-8a7c-13f2a0abaa74\" (UID: \"efe7c063-fb0e-4679-8a7c-13f2a0abaa74\") "
Oct 30 22:22:51.881835 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:51.881750    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efe7c063-fb0e-4679-8a7c-13f2a0abaa74-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/efe7c063-fb0e-4679-8a7c-13f2a0abaa74-tmp-dir\") pod \"efe7c063-fb0e-4679-8a7c-13f2a0abaa74\" (UID: \"efe7c063-fb0e-4679-8a7c-13f2a0abaa74\") "
Oct 30 22:22:51.881835 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:51.881783    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/efe7c063-fb0e-4679-8a7c-13f2a0abaa74-tls-certs\" (UniqueName: \"kubernetes.io/secret/efe7c063-fb0e-4679-8a7c-13f2a0abaa74-tls-certs\") pod \"efe7c063-fb0e-4679-8a7c-13f2a0abaa74\" (UID: \"efe7c063-fb0e-4679-8a7c-13f2a0abaa74\") "
Oct 30 22:22:51.881959 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:51.881873    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efe7c063-fb0e-4679-8a7c-13f2a0abaa74-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/efe7c063-fb0e-4679-8a7c-13f2a0abaa74-model-cache\") pod \"efe7c063-fb0e-4679-8a7c-13f2a0abaa74\" (UID: \"efe7c063-fb0e-4679-8a7c-13f2a0abaa74\") "
Oct 30 22:22:51.881959 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:51.881924    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efe7c063-fb0e-4679-8a7c-13f2a0abaa74-kube-api-access-4r2fq\" (UniqueName: \"kubernetes.io/projected/efe7c063-fb0e-4679-8a7c-13f2a0abaa74-kube-api-access-4r2fq\") pod \"efe7c063-fb0e-4679-8a7c-13f2a0abaa74\" (UID: \"efe7c063-fb0e-4679-8a7c-13f2a0abaa74\") "
Oct 30 22:22:51.882432 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:51.882408    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efe7c063-fb0e-4679-8a7c-13f2a0abaa74-model-cache" pod "efe7c063-fb0e-4679-8a7c-13f2a0abaa74" (UID: "efe7c063-fb0e-4679-8a7c-13f2a0abaa74"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:22:51.882583 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:51.882490    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efe7c063-fb0e-4679-8a7c-13f2a0abaa74-tmp-dir" pod "efe7c063-fb0e-4679-8a7c-13f2a0abaa74" (UID: "efe7c063-fb0e-4679-8a7c-13f2a0abaa74"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:22:51.882669 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:51.882595    5061 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/efe7c063-fb0e-4679-8a7c-13f2a0abaa74-home\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:51.884782 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:51.884756    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efe7c063-fb0e-4679-8a7c-13f2a0abaa74-dshm" pod "efe7c063-fb0e-4679-8a7c-13f2a0abaa74" (UID: "efe7c063-fb0e-4679-8a7c-13f2a0abaa74"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:22:51.886597 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:51.886567    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efe7c063-fb0e-4679-8a7c-13f2a0abaa74-kube-api-access-4r2fq" pod "efe7c063-fb0e-4679-8a7c-13f2a0abaa74" (UID: "efe7c063-fb0e-4679-8a7c-13f2a0abaa74"). InnerVolumeSpecName "kube-api-access-4r2fq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:22:51.886710 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:51.886601    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efe7c063-fb0e-4679-8a7c-13f2a0abaa74-tls-certs" pod "efe7c063-fb0e-4679-8a7c-13f2a0abaa74" (UID: "efe7c063-fb0e-4679-8a7c-13f2a0abaa74"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:22:51.962815 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:51.962512    5061 generic.go:356] "Generic (PLEG): container finished" podID="efe7c063-fb0e-4679-8a7c-13f2a0abaa74" containerID="0feba69f7cea4a1d44ba150288760dd74da547313ff80433eec1bf73c2b9c466" exitCode=0
Oct 30 22:22:51.969151 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:51.963075    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a07620/simulator-w1-a07620-kserve-55b44674f4-4z9lk"
Oct 30 22:22:51.969151 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:51.966905    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a07620/simulator-w1-a07620-kserve-55b44674f4-4z9lk" event={"ID":"efe7c063-fb0e-4679-8a7c-13f2a0abaa74","Type":"ContainerDied","Data":"0feba69f7cea4a1d44ba150288760dd74da547313ff80433eec1bf73c2b9c466"}
Oct 30 22:22:51.969151 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:51.966952    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a07620/simulator-w1-a07620-kserve-55b44674f4-4z9lk" event={"ID":"efe7c063-fb0e-4679-8a7c-13f2a0abaa74","Type":"ContainerDied","Data":"a0e4fbd8ee4d11a80962c68d4c12d5f3fa1d364f6aba042490ad79e80412fbd2"}
Oct 30 22:22:51.969151 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:51.966975    5061 scope.go:122] "RemoveContainer" containerID="0feba69f7cea4a1d44ba150288760dd74da547313ff80433eec1bf73c2b9c466"
Oct 30 22:22:51.984298 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:51.984274    5061 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/efe7c063-fb0e-4679-8a7c-13f2a0abaa74-model-cache\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:51.984298 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:51.984299    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4r2fq\" (UniqueName: \"kubernetes.io/projected/efe7c063-fb0e-4679-8a7c-13f2a0abaa74-kube-api-access-4r2fq\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:51.984437 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:51.984308    5061 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/efe7c063-fb0e-4679-8a7c-13f2a0abaa74-dshm\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:51.984437 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:51.984317    5061 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/efe7c063-fb0e-4679-8a7c-13f2a0abaa74-tmp-dir\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:51.984437 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:51.984325    5061 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efe7c063-fb0e-4679-8a7c-13f2a0abaa74-tls-certs\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:51.987130 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:51.987111    5061 scope.go:122] "RemoveContainer" containerID="0feba69f7cea4a1d44ba150288760dd74da547313ff80433eec1bf73c2b9c466"
Oct 30 22:22:51.987408 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:22:51.987390    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0feba69f7cea4a1d44ba150288760dd74da547313ff80433eec1bf73c2b9c466\": container with ID starting with 0feba69f7cea4a1d44ba150288760dd74da547313ff80433eec1bf73c2b9c466 not found: ID does not exist" containerID="0feba69f7cea4a1d44ba150288760dd74da547313ff80433eec1bf73c2b9c466"
Oct 30 22:22:51.987473 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:51.987414    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0feba69f7cea4a1d44ba150288760dd74da547313ff80433eec1bf73c2b9c466"} err="failed to get container status \"0feba69f7cea4a1d44ba150288760dd74da547313ff80433eec1bf73c2b9c466\": rpc error: code = NotFound desc = could not find container \"0feba69f7cea4a1d44ba150288760dd74da547313ff80433eec1bf73c2b9c466\": container with ID starting with 0feba69f7cea4a1d44ba150288760dd74da547313ff80433eec1bf73c2b9c466 not found: ID does not exist"
Oct 30 22:22:52.137915 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:52.137876    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-a07620/simulator-w1-a07620-kserve-55b44674f4-4z9lk"]
Oct 30 22:22:52.142415 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:52.142389    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-a07620/simulator-w1-a07620-kserve-55b44674f4-4z9lk"]
Oct 30 22:22:52.543830 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:52.543799    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="efe7c063-fb0e-4679-8a7c-13f2a0abaa74" path="/var/lib/kubelet/pods/efe7c063-fb0e-4679-8a7c-13f2a0abaa74/volumes"
Oct 30 22:22:53.233884 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:53.233846    5061 patch_prober.go:29] interesting pod/e2e-mt-778d39d3-openshift-default-bff74755c-9r5c8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:22:53.234061 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:53.233903    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-9r5c8" podUID="83adee7e-3585-4d0d-b1b2-d4b5d3be0588" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:22:53.789038 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:53.789011    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-9r5c8"
Oct 30 22:22:53.843991 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:53.838029    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83adee7e-3585-4d0d-b1b2-d4b5d3be0588-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83adee7e-3585-4d0d-b1b2-d4b5d3be0588-istio-data\") pod \"83adee7e-3585-4d0d-b1b2-d4b5d3be0588\" (UID: \"83adee7e-3585-4d0d-b1b2-d4b5d3be0588\") "
Oct 30 22:22:53.843991 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:53.838092    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83adee7e-3585-4d0d-b1b2-d4b5d3be0588-istio-token\" (UniqueName: \"kubernetes.io/projected/83adee7e-3585-4d0d-b1b2-d4b5d3be0588-istio-token\") pod \"83adee7e-3585-4d0d-b1b2-d4b5d3be0588\" (UID: \"83adee7e-3585-4d0d-b1b2-d4b5d3be0588\") "
Oct 30 22:22:53.843991 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:53.838183    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/83adee7e-3585-4d0d-b1b2-d4b5d3be0588-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83adee7e-3585-4d0d-b1b2-d4b5d3be0588-istio-podinfo\") pod \"83adee7e-3585-4d0d-b1b2-d4b5d3be0588\" (UID: \"83adee7e-3585-4d0d-b1b2-d4b5d3be0588\") "
Oct 30 22:22:53.843991 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:53.838228    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83adee7e-3585-4d0d-b1b2-d4b5d3be0588-kube-api-access-gqs79\" (UniqueName: \"kubernetes.io/projected/83adee7e-3585-4d0d-b1b2-d4b5d3be0588-kube-api-access-gqs79\") pod \"83adee7e-3585-4d0d-b1b2-d4b5d3be0588\" (UID: \"83adee7e-3585-4d0d-b1b2-d4b5d3be0588\") "
Oct 30 22:22:53.843991 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:53.838294    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83adee7e-3585-4d0d-b1b2-d4b5d3be0588-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83adee7e-3585-4d0d-b1b2-d4b5d3be0588-istio-envoy\") pod \"83adee7e-3585-4d0d-b1b2-d4b5d3be0588\" (UID: \"83adee7e-3585-4d0d-b1b2-d4b5d3be0588\") "
Oct 30 22:22:53.843991 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:53.838340    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83adee7e-3585-4d0d-b1b2-d4b5d3be0588-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83adee7e-3585-4d0d-b1b2-d4b5d3be0588-istiod-ca-cert\") pod \"83adee7e-3585-4d0d-b1b2-d4b5d3be0588\" (UID: \"83adee7e-3585-4d0d-b1b2-d4b5d3be0588\") "
Oct 30 22:22:53.843991 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:53.838565    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83adee7e-3585-4d0d-b1b2-d4b5d3be0588-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83adee7e-3585-4d0d-b1b2-d4b5d3be0588-credential-socket\") pod \"83adee7e-3585-4d0d-b1b2-d4b5d3be0588\" (UID: \"83adee7e-3585-4d0d-b1b2-d4b5d3be0588\") "
Oct 30 22:22:53.843991 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:53.838602    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83adee7e-3585-4d0d-b1b2-d4b5d3be0588-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83adee7e-3585-4d0d-b1b2-d4b5d3be0588-workload-certs\") pod \"83adee7e-3585-4d0d-b1b2-d4b5d3be0588\" (UID: \"83adee7e-3585-4d0d-b1b2-d4b5d3be0588\") "
Oct 30 22:22:53.843991 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:53.838659    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83adee7e-3585-4d0d-b1b2-d4b5d3be0588-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83adee7e-3585-4d0d-b1b2-d4b5d3be0588-workload-socket\") pod \"83adee7e-3585-4d0d-b1b2-d4b5d3be0588\" (UID: \"83adee7e-3585-4d0d-b1b2-d4b5d3be0588\") "
Oct 30 22:22:53.843991 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:53.839778    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83adee7e-3585-4d0d-b1b2-d4b5d3be0588-workload-socket" pod "83adee7e-3585-4d0d-b1b2-d4b5d3be0588" (UID: "83adee7e-3585-4d0d-b1b2-d4b5d3be0588"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:22:53.843991 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:53.839905    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83adee7e-3585-4d0d-b1b2-d4b5d3be0588-istio-data" pod "83adee7e-3585-4d0d-b1b2-d4b5d3be0588" (UID: "83adee7e-3585-4d0d-b1b2-d4b5d3be0588"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:22:53.843991 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:53.840408    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83adee7e-3585-4d0d-b1b2-d4b5d3be0588-workload-certs" pod "83adee7e-3585-4d0d-b1b2-d4b5d3be0588" (UID: "83adee7e-3585-4d0d-b1b2-d4b5d3be0588"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:22:53.843991 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:53.840464    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83adee7e-3585-4d0d-b1b2-d4b5d3be0588-credential-socket" pod "83adee7e-3585-4d0d-b1b2-d4b5d3be0588" (UID: "83adee7e-3585-4d0d-b1b2-d4b5d3be0588"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:22:53.843991 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:53.840706    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83adee7e-3585-4d0d-b1b2-d4b5d3be0588-istiod-ca-cert" pod "83adee7e-3585-4d0d-b1b2-d4b5d3be0588" (UID: "83adee7e-3585-4d0d-b1b2-d4b5d3be0588"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:22:53.843991 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:53.842586    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83adee7e-3585-4d0d-b1b2-d4b5d3be0588-kube-api-access-gqs79" pod "83adee7e-3585-4d0d-b1b2-d4b5d3be0588" (UID: "83adee7e-3585-4d0d-b1b2-d4b5d3be0588"). InnerVolumeSpecName "kube-api-access-gqs79". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:22:53.845050 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:53.844498    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83adee7e-3585-4d0d-b1b2-d4b5d3be0588-istio-token" pod "83adee7e-3585-4d0d-b1b2-d4b5d3be0588" (UID: "83adee7e-3585-4d0d-b1b2-d4b5d3be0588"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:22:53.845463 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:53.845439    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83adee7e-3585-4d0d-b1b2-d4b5d3be0588-istio-envoy" pod "83adee7e-3585-4d0d-b1b2-d4b5d3be0588" (UID: "83adee7e-3585-4d0d-b1b2-d4b5d3be0588"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:22:53.845626 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:53.845490    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/83adee7e-3585-4d0d-b1b2-d4b5d3be0588-istio-podinfo" pod "83adee7e-3585-4d0d-b1b2-d4b5d3be0588" (UID: "83adee7e-3585-4d0d-b1b2-d4b5d3be0588"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:22:53.871877 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:53.871847    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-k5knf"
Oct 30 22:22:53.896904 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:53.896879    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-nv6kl"
Oct 30 22:22:53.941061 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:53.941025    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/732b6214-aa8c-45fc-bd06-c0f75a267960-istio-token\" (UniqueName: \"kubernetes.io/projected/732b6214-aa8c-45fc-bd06-c0f75a267960-istio-token\") pod \"732b6214-aa8c-45fc-bd06-c0f75a267960\" (UID: \"732b6214-aa8c-45fc-bd06-c0f75a267960\") "
Oct 30 22:22:53.941219 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:53.941118    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3489262b-293e-4dba-8bb2-efa4333ab66d-istio-token\" (UniqueName: \"kubernetes.io/projected/3489262b-293e-4dba-8bb2-efa4333ab66d-istio-token\") pod \"3489262b-293e-4dba-8bb2-efa4333ab66d\" (UID: \"3489262b-293e-4dba-8bb2-efa4333ab66d\") "
Oct 30 22:22:53.941219 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:53.941162    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/732b6214-aa8c-45fc-bd06-c0f75a267960-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/732b6214-aa8c-45fc-bd06-c0f75a267960-istio-envoy\") pod \"732b6214-aa8c-45fc-bd06-c0f75a267960\" (UID: \"732b6214-aa8c-45fc-bd06-c0f75a267960\") "
Oct 30 22:22:53.941219 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:53.941191    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/732b6214-aa8c-45fc-bd06-c0f75a267960-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/732b6214-aa8c-45fc-bd06-c0f75a267960-credential-socket\") pod \"732b6214-aa8c-45fc-bd06-c0f75a267960\" (UID: \"732b6214-aa8c-45fc-bd06-c0f75a267960\") "
Oct 30 22:22:53.941412 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:53.941252    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3489262b-293e-4dba-8bb2-efa4333ab66d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3489262b-293e-4dba-8bb2-efa4333ab66d-workload-certs\") pod \"3489262b-293e-4dba-8bb2-efa4333ab66d\" (UID: \"3489262b-293e-4dba-8bb2-efa4333ab66d\") "
Oct 30 22:22:53.941412 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:53.941281    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3489262b-293e-4dba-8bb2-efa4333ab66d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3489262b-293e-4dba-8bb2-efa4333ab66d-istiod-ca-cert\") pod \"3489262b-293e-4dba-8bb2-efa4333ab66d\" (UID: \"3489262b-293e-4dba-8bb2-efa4333ab66d\") "
Oct 30 22:22:53.943265 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:53.943153    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/732b6214-aa8c-45fc-bd06-c0f75a267960-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/732b6214-aa8c-45fc-bd06-c0f75a267960-workload-socket\") pod \"732b6214-aa8c-45fc-bd06-c0f75a267960\" (UID: \"732b6214-aa8c-45fc-bd06-c0f75a267960\") "
Oct 30 22:22:53.943265 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:53.943205    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/732b6214-aa8c-45fc-bd06-c0f75a267960-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/732b6214-aa8c-45fc-bd06-c0f75a267960-workload-certs\") pod \"732b6214-aa8c-45fc-bd06-c0f75a267960\" (UID: \"732b6214-aa8c-45fc-bd06-c0f75a267960\") "
Oct 30 22:22:53.943265 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:53.943254    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/732b6214-aa8c-45fc-bd06-c0f75a267960-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/732b6214-aa8c-45fc-bd06-c0f75a267960-istio-data\") pod \"732b6214-aa8c-45fc-bd06-c0f75a267960\" (UID: \"732b6214-aa8c-45fc-bd06-c0f75a267960\") "
Oct 30 22:22:53.943440 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:53.943286    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/732b6214-aa8c-45fc-bd06-c0f75a267960-kube-api-access-v9kmg\" (UniqueName: \"kubernetes.io/projected/732b6214-aa8c-45fc-bd06-c0f75a267960-kube-api-access-v9kmg\") pod \"732b6214-aa8c-45fc-bd06-c0f75a267960\" (UID: \"732b6214-aa8c-45fc-bd06-c0f75a267960\") "
Oct 30 22:22:53.943440 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:53.943390    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/732b6214-aa8c-45fc-bd06-c0f75a267960-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/732b6214-aa8c-45fc-bd06-c0f75a267960-istiod-ca-cert\") pod \"732b6214-aa8c-45fc-bd06-c0f75a267960\" (UID: \"732b6214-aa8c-45fc-bd06-c0f75a267960\") "
Oct 30 22:22:53.943519 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:53.943468    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3489262b-293e-4dba-8bb2-efa4333ab66d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3489262b-293e-4dba-8bb2-efa4333ab66d-istio-envoy\") pod \"3489262b-293e-4dba-8bb2-efa4333ab66d\" (UID: \"3489262b-293e-4dba-8bb2-efa4333ab66d\") "
Oct 30 22:22:53.943571 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:53.943542    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/732b6214-aa8c-45fc-bd06-c0f75a267960-credential-socket" pod "732b6214-aa8c-45fc-bd06-c0f75a267960" (UID: "732b6214-aa8c-45fc-bd06-c0f75a267960"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:22:53.943625 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:53.943564    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3489262b-293e-4dba-8bb2-efa4333ab66d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3489262b-293e-4dba-8bb2-efa4333ab66d-workload-socket\") pod \"3489262b-293e-4dba-8bb2-efa4333ab66d\" (UID: \"3489262b-293e-4dba-8bb2-efa4333ab66d\") "
Oct 30 22:22:53.943625 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:53.943620    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3489262b-293e-4dba-8bb2-efa4333ab66d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3489262b-293e-4dba-8bb2-efa4333ab66d-istio-data\") pod \"3489262b-293e-4dba-8bb2-efa4333ab66d\" (UID: \"3489262b-293e-4dba-8bb2-efa4333ab66d\") "
Oct 30 22:22:53.943737 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:53.943658    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/732b6214-aa8c-45fc-bd06-c0f75a267960-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/732b6214-aa8c-45fc-bd06-c0f75a267960-istio-podinfo\") pod \"732b6214-aa8c-45fc-bd06-c0f75a267960\" (UID: \"732b6214-aa8c-45fc-bd06-c0f75a267960\") "
Oct 30 22:22:53.943737 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:53.943701    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3489262b-293e-4dba-8bb2-efa4333ab66d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3489262b-293e-4dba-8bb2-efa4333ab66d-istio-podinfo\") pod \"3489262b-293e-4dba-8bb2-efa4333ab66d\" (UID: \"3489262b-293e-4dba-8bb2-efa4333ab66d\") "
Oct 30 22:22:53.943842 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:53.943741    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3489262b-293e-4dba-8bb2-efa4333ab66d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3489262b-293e-4dba-8bb2-efa4333ab66d-credential-socket\") pod \"3489262b-293e-4dba-8bb2-efa4333ab66d\" (UID: \"3489262b-293e-4dba-8bb2-efa4333ab66d\") "
Oct 30 22:22:53.943842 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:53.943777    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3489262b-293e-4dba-8bb2-efa4333ab66d-kube-api-access-9dg4j\" (UniqueName: \"kubernetes.io/projected/3489262b-293e-4dba-8bb2-efa4333ab66d-kube-api-access-9dg4j\") pod \"3489262b-293e-4dba-8bb2-efa4333ab66d\" (UID: \"3489262b-293e-4dba-8bb2-efa4333ab66d\") "
Oct 30 22:22:53.944505 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:53.944480    5061 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83adee7e-3585-4d0d-b1b2-d4b5d3be0588-workload-socket\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:53.944629 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:53.944507    5061 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83adee7e-3585-4d0d-b1b2-d4b5d3be0588-istio-data\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:53.944629 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:53.944537    5061 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83adee7e-3585-4d0d-b1b2-d4b5d3be0588-istio-token\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:53.944629 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:53.944552    5061 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/732b6214-aa8c-45fc-bd06-c0f75a267960-credential-socket\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:53.944629 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:53.944569    5061 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83adee7e-3585-4d0d-b1b2-d4b5d3be0588-istio-podinfo\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:53.944629 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:53.944583    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqs79\" (UniqueName: \"kubernetes.io/projected/83adee7e-3585-4d0d-b1b2-d4b5d3be0588-kube-api-access-gqs79\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:53.944629 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:53.944598    5061 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83adee7e-3585-4d0d-b1b2-d4b5d3be0588-istio-envoy\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:53.944629 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:53.944613    5061 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83adee7e-3585-4d0d-b1b2-d4b5d3be0588-istiod-ca-cert\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:53.944629 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:53.944627    5061 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83adee7e-3585-4d0d-b1b2-d4b5d3be0588-credential-socket\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:53.945053 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:53.944641    5061 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83adee7e-3585-4d0d-b1b2-d4b5d3be0588-workload-certs\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:53.945053 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:53.944704    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3489262b-293e-4dba-8bb2-efa4333ab66d-istiod-ca-cert" pod "3489262b-293e-4dba-8bb2-efa4333ab66d" (UID: "3489262b-293e-4dba-8bb2-efa4333ab66d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:22:53.945053 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:53.944964    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/732b6214-aa8c-45fc-bd06-c0f75a267960-istio-data" pod "732b6214-aa8c-45fc-bd06-c0f75a267960" (UID: "732b6214-aa8c-45fc-bd06-c0f75a267960"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:22:53.945208 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:53.945136    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/732b6214-aa8c-45fc-bd06-c0f75a267960-workload-socket" pod "732b6214-aa8c-45fc-bd06-c0f75a267960" (UID: "732b6214-aa8c-45fc-bd06-c0f75a267960"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:22:53.945549 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:53.945299    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/732b6214-aa8c-45fc-bd06-c0f75a267960-workload-certs" pod "732b6214-aa8c-45fc-bd06-c0f75a267960" (UID: "732b6214-aa8c-45fc-bd06-c0f75a267960"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:22:53.945663 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:53.945583    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3489262b-293e-4dba-8bb2-efa4333ab66d-workload-certs" pod "3489262b-293e-4dba-8bb2-efa4333ab66d" (UID: "3489262b-293e-4dba-8bb2-efa4333ab66d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:22:53.949550 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:53.949039    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/732b6214-aa8c-45fc-bd06-c0f75a267960-istio-envoy" pod "732b6214-aa8c-45fc-bd06-c0f75a267960" (UID: "732b6214-aa8c-45fc-bd06-c0f75a267960"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:22:53.949550 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:53.949144    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3489262b-293e-4dba-8bb2-efa4333ab66d-kube-api-access-9dg4j" pod "3489262b-293e-4dba-8bb2-efa4333ab66d" (UID: "3489262b-293e-4dba-8bb2-efa4333ab66d"). InnerVolumeSpecName "kube-api-access-9dg4j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:22:53.951507 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:53.951355    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/732b6214-aa8c-45fc-bd06-c0f75a267960-istio-token" pod "732b6214-aa8c-45fc-bd06-c0f75a267960" (UID: "732b6214-aa8c-45fc-bd06-c0f75a267960"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:22:53.951507 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:53.951320    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3489262b-293e-4dba-8bb2-efa4333ab66d-workload-socket" pod "3489262b-293e-4dba-8bb2-efa4333ab66d" (UID: "3489262b-293e-4dba-8bb2-efa4333ab66d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:22:53.951635 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:53.951476    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/732b6214-aa8c-45fc-bd06-c0f75a267960-istiod-ca-cert" pod "732b6214-aa8c-45fc-bd06-c0f75a267960" (UID: "732b6214-aa8c-45fc-bd06-c0f75a267960"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:22:53.951772 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:53.951705    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3489262b-293e-4dba-8bb2-efa4333ab66d-credential-socket" pod "3489262b-293e-4dba-8bb2-efa4333ab66d" (UID: "3489262b-293e-4dba-8bb2-efa4333ab66d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:22:53.951772 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:53.951740    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3489262b-293e-4dba-8bb2-efa4333ab66d-istio-data" pod "3489262b-293e-4dba-8bb2-efa4333ab66d" (UID: "3489262b-293e-4dba-8bb2-efa4333ab66d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:22:53.955962 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:53.953075    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3489262b-293e-4dba-8bb2-efa4333ab66d-istio-envoy" pod "3489262b-293e-4dba-8bb2-efa4333ab66d" (UID: "3489262b-293e-4dba-8bb2-efa4333ab66d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:22:53.955962 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:53.953394    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3489262b-293e-4dba-8bb2-efa4333ab66d-istio-token" pod "3489262b-293e-4dba-8bb2-efa4333ab66d" (UID: "3489262b-293e-4dba-8bb2-efa4333ab66d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:22:53.955962 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:53.953624    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/732b6214-aa8c-45fc-bd06-c0f75a267960-istio-podinfo" pod "732b6214-aa8c-45fc-bd06-c0f75a267960" (UID: "732b6214-aa8c-45fc-bd06-c0f75a267960"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:22:53.955962 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:53.953806    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/732b6214-aa8c-45fc-bd06-c0f75a267960-kube-api-access-v9kmg" pod "732b6214-aa8c-45fc-bd06-c0f75a267960" (UID: "732b6214-aa8c-45fc-bd06-c0f75a267960"). InnerVolumeSpecName "kube-api-access-v9kmg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:22:53.955962 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:53.954030    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3489262b-293e-4dba-8bb2-efa4333ab66d-istio-podinfo" pod "3489262b-293e-4dba-8bb2-efa4333ab66d" (UID: "3489262b-293e-4dba-8bb2-efa4333ab66d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:22:53.970573 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:53.962652    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-649qh"
Oct 30 22:22:53.987082 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:53.987055    5061 generic.go:356] "Generic (PLEG): container finished" podID="83adee7e-3585-4d0d-b1b2-d4b5d3be0588" containerID="6a6cc21e10bc1e0353ea3b36fe7f807ea2a367e761da2d139b8e0f3b8009c1f0" exitCode=0
Oct 30 22:22:53.987261 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:53.987123    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-9r5c8"
Oct 30 22:22:53.987261 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:53.987140    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-9r5c8" event={"ID":"83adee7e-3585-4d0d-b1b2-d4b5d3be0588","Type":"ContainerDied","Data":"6a6cc21e10bc1e0353ea3b36fe7f807ea2a367e761da2d139b8e0f3b8009c1f0"}
Oct 30 22:22:53.987261 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:53.987178    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-9r5c8" event={"ID":"83adee7e-3585-4d0d-b1b2-d4b5d3be0588","Type":"ContainerDied","Data":"311caf6c228b1dd09e6a7ae2a7478c8481c1993891207439288228ffc36c5123"}
Oct 30 22:22:53.987261 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:53.987208    5061 scope.go:122] "RemoveContainer" containerID="6a6cc21e10bc1e0353ea3b36fe7f807ea2a367e761da2d139b8e0f3b8009c1f0"
Oct 30 22:22:53.995332 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:53.995301    5061 generic.go:356] "Generic (PLEG): container finished" podID="3489262b-293e-4dba-8bb2-efa4333ab66d" containerID="69650f186337a304c285b07f94defc757c4f41455ed19bfc01752e056d037b6c" exitCode=0
Oct 30 22:22:53.995452 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:53.995335    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-k5knf" event={"ID":"3489262b-293e-4dba-8bb2-efa4333ab66d","Type":"ContainerDied","Data":"69650f186337a304c285b07f94defc757c4f41455ed19bfc01752e056d037b6c"}
Oct 30 22:22:53.995452 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:53.995385    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-k5knf" event={"ID":"3489262b-293e-4dba-8bb2-efa4333ab66d","Type":"ContainerDied","Data":"6b537cdbf17f3ddd2abc593fd2fd5b51dff360bcf94560929176170c069d8cfa"}
Oct 30 22:22:53.995783 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:53.995761    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-k5knf"
Oct 30 22:22:53.998480 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:53.997053    5061 generic.go:356] "Generic (PLEG): container finished" podID="732b6214-aa8c-45fc-bd06-c0f75a267960" containerID="9cb0b7e5f37d1b429283bcdb104977b0c36564f7cf0cc9f930c4493a4c93fe50" exitCode=0
Oct 30 22:22:53.998480 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:53.997169    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-nv6kl" event={"ID":"732b6214-aa8c-45fc-bd06-c0f75a267960","Type":"ContainerDied","Data":"9cb0b7e5f37d1b429283bcdb104977b0c36564f7cf0cc9f930c4493a4c93fe50"}
Oct 30 22:22:53.998480 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:53.997187    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-nv6kl"
Oct 30 22:22:53.998480 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:53.997218    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-nv6kl" event={"ID":"732b6214-aa8c-45fc-bd06-c0f75a267960","Type":"ContainerDied","Data":"f2550a67274b63f77056755c091d44a7944829a5f0bd0a35508871ab93a55ced"}
Oct 30 22:22:54.004778 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:53.999558    5061 generic.go:356] "Generic (PLEG): container finished" podID="598138a5-2e71-4727-9332-3c604144f0ef" containerID="b38d0702c46a0f99ecacb9cbe7ed469a5bd18364eb5845fc32519c06825cc329" exitCode=0
Oct 30 22:22:54.004778 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:53.999764    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-649qh" event={"ID":"598138a5-2e71-4727-9332-3c604144f0ef","Type":"ContainerDied","Data":"b38d0702c46a0f99ecacb9cbe7ed469a5bd18364eb5845fc32519c06825cc329"}
Oct 30 22:22:54.004778 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:53.999803    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-649qh" event={"ID":"598138a5-2e71-4727-9332-3c604144f0ef","Type":"ContainerDied","Data":"0684a5651d0c6871a33081643ba48d3a9af9bbcab8136b520a37de84d316affd"}
Oct 30 22:22:54.004778 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:53.999872    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-649qh"
Oct 30 22:22:54.039195 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:54.039169    5061 scope.go:122] "RemoveContainer" containerID="28d948b4308d1a2ef8191c64f52379ad5384f4efea11fb8817b380e34d76df2d"
Oct 30 22:22:54.046349 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:54.046327    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/598138a5-2e71-4727-9332-3c604144f0ef-kube-api-access-6zkwn\" (UniqueName: \"kubernetes.io/projected/598138a5-2e71-4727-9332-3c604144f0ef-kube-api-access-6zkwn\") pod \"598138a5-2e71-4727-9332-3c604144f0ef\" (UID: \"598138a5-2e71-4727-9332-3c604144f0ef\") "
Oct 30 22:22:54.046596 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:54.046573    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/598138a5-2e71-4727-9332-3c604144f0ef-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/598138a5-2e71-4727-9332-3c604144f0ef-istiod-ca-cert\") pod \"598138a5-2e71-4727-9332-3c604144f0ef\" (UID: \"598138a5-2e71-4727-9332-3c604144f0ef\") "
Oct 30 22:22:54.046892 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:54.046871    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/598138a5-2e71-4727-9332-3c604144f0ef-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/598138a5-2e71-4727-9332-3c604144f0ef-istio-envoy\") pod \"598138a5-2e71-4727-9332-3c604144f0ef\" (UID: \"598138a5-2e71-4727-9332-3c604144f0ef\") "
Oct 30 22:22:54.047096 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:54.047063    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/598138a5-2e71-4727-9332-3c604144f0ef-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/598138a5-2e71-4727-9332-3c604144f0ef-workload-socket\") pod \"598138a5-2e71-4727-9332-3c604144f0ef\" (UID: \"598138a5-2e71-4727-9332-3c604144f0ef\") "
Oct 30 22:22:54.047299 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:54.047245    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/598138a5-2e71-4727-9332-3c604144f0ef-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/598138a5-2e71-4727-9332-3c604144f0ef-workload-certs\") pod \"598138a5-2e71-4727-9332-3c604144f0ef\" (UID: \"598138a5-2e71-4727-9332-3c604144f0ef\") "
Oct 30 22:22:54.047538 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:54.047494    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/598138a5-2e71-4727-9332-3c604144f0ef-workload-certs" pod "598138a5-2e71-4727-9332-3c604144f0ef" (UID: "598138a5-2e71-4727-9332-3c604144f0ef"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:22:54.047789 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:54.047618    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/598138a5-2e71-4727-9332-3c604144f0ef-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/598138a5-2e71-4727-9332-3c604144f0ef-credential-socket\") pod \"598138a5-2e71-4727-9332-3c604144f0ef\" (UID: \"598138a5-2e71-4727-9332-3c604144f0ef\") "
Oct 30 22:22:54.047789 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:54.047767    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/598138a5-2e71-4727-9332-3c604144f0ef-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/598138a5-2e71-4727-9332-3c604144f0ef-istio-podinfo\") pod \"598138a5-2e71-4727-9332-3c604144f0ef\" (UID: \"598138a5-2e71-4727-9332-3c604144f0ef\") "
Oct 30 22:22:54.047937 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:54.047805    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/598138a5-2e71-4727-9332-3c604144f0ef-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/598138a5-2e71-4727-9332-3c604144f0ef-istio-data\") pod \"598138a5-2e71-4727-9332-3c604144f0ef\" (UID: \"598138a5-2e71-4727-9332-3c604144f0ef\") "
Oct 30 22:22:54.048007 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:54.047958    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/598138a5-2e71-4727-9332-3c604144f0ef-istio-token\" (UniqueName: \"kubernetes.io/projected/598138a5-2e71-4727-9332-3c604144f0ef-istio-token\") pod \"598138a5-2e71-4727-9332-3c604144f0ef\" (UID: \"598138a5-2e71-4727-9332-3c604144f0ef\") "
Oct 30 22:22:54.049756 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:54.048740    5061 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/598138a5-2e71-4727-9332-3c604144f0ef-workload-certs\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:54.049756 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:54.048766    5061 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/732b6214-aa8c-45fc-bd06-c0f75a267960-istio-token\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:54.049756 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:54.048799    5061 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3489262b-293e-4dba-8bb2-efa4333ab66d-istio-token\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:54.049756 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:54.048812    5061 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/732b6214-aa8c-45fc-bd06-c0f75a267960-istio-envoy\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:54.049756 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:54.048827    5061 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3489262b-293e-4dba-8bb2-efa4333ab66d-workload-certs\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:54.049756 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:54.048840    5061 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3489262b-293e-4dba-8bb2-efa4333ab66d-istiod-ca-cert\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:54.049756 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:54.048868    5061 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/732b6214-aa8c-45fc-bd06-c0f75a267960-workload-socket\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:54.049756 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:54.048882    5061 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/732b6214-aa8c-45fc-bd06-c0f75a267960-workload-certs\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:54.049756 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:54.048895    5061 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/732b6214-aa8c-45fc-bd06-c0f75a267960-istio-data\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:54.049756 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:54.048913    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9kmg\" (UniqueName: \"kubernetes.io/projected/732b6214-aa8c-45fc-bd06-c0f75a267960-kube-api-access-v9kmg\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:54.049756 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:54.048942    5061 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/732b6214-aa8c-45fc-bd06-c0f75a267960-istiod-ca-cert\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:54.049756 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:54.048957    5061 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3489262b-293e-4dba-8bb2-efa4333ab66d-istio-envoy\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:54.049756 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:54.048970    5061 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3489262b-293e-4dba-8bb2-efa4333ab66d-workload-socket\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:54.049756 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:54.048985    5061 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3489262b-293e-4dba-8bb2-efa4333ab66d-istio-data\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:54.049756 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:54.048998    5061 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/732b6214-aa8c-45fc-bd06-c0f75a267960-istio-podinfo\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:54.049756 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:54.049027    5061 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3489262b-293e-4dba-8bb2-efa4333ab66d-istio-podinfo\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:54.049756 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:54.049041    5061 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3489262b-293e-4dba-8bb2-efa4333ab66d-credential-socket\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:54.049756 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:54.049056    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dg4j\" (UniqueName: \"kubernetes.io/projected/3489262b-293e-4dba-8bb2-efa4333ab66d-kube-api-access-9dg4j\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:54.049756 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:54.049624    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/598138a5-2e71-4727-9332-3c604144f0ef-workload-socket" pod "598138a5-2e71-4727-9332-3c604144f0ef" (UID: "598138a5-2e71-4727-9332-3c604144f0ef"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:22:54.051060 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:54.049877    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/598138a5-2e71-4727-9332-3c604144f0ef-istiod-ca-cert" pod "598138a5-2e71-4727-9332-3c604144f0ef" (UID: "598138a5-2e71-4727-9332-3c604144f0ef"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:22:54.051060 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:54.050426    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/598138a5-2e71-4727-9332-3c604144f0ef-istio-data" pod "598138a5-2e71-4727-9332-3c604144f0ef" (UID: "598138a5-2e71-4727-9332-3c604144f0ef"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:22:54.051060 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:54.050486    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/598138a5-2e71-4727-9332-3c604144f0ef-credential-socket" pod "598138a5-2e71-4727-9332-3c604144f0ef" (UID: "598138a5-2e71-4727-9332-3c604144f0ef"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:22:54.055326 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:54.054677    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/598138a5-2e71-4727-9332-3c604144f0ef-istio-envoy" pod "598138a5-2e71-4727-9332-3c604144f0ef" (UID: "598138a5-2e71-4727-9332-3c604144f0ef"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:22:54.055326 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:54.055123    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/598138a5-2e71-4727-9332-3c604144f0ef-istio-podinfo" pod "598138a5-2e71-4727-9332-3c604144f0ef" (UID: "598138a5-2e71-4727-9332-3c604144f0ef"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:22:54.055438 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:54.055357    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/598138a5-2e71-4727-9332-3c604144f0ef-istio-token" pod "598138a5-2e71-4727-9332-3c604144f0ef" (UID: "598138a5-2e71-4727-9332-3c604144f0ef"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:22:54.056521 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:54.056494    5061 scope.go:122] "RemoveContainer" containerID="6a6cc21e10bc1e0353ea3b36fe7f807ea2a367e761da2d139b8e0f3b8009c1f0"
Oct 30 22:22:54.056909 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:22:54.056883    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a6cc21e10bc1e0353ea3b36fe7f807ea2a367e761da2d139b8e0f3b8009c1f0\": container with ID starting with 6a6cc21e10bc1e0353ea3b36fe7f807ea2a367e761da2d139b8e0f3b8009c1f0 not found: ID does not exist" containerID="6a6cc21e10bc1e0353ea3b36fe7f807ea2a367e761da2d139b8e0f3b8009c1f0"
Oct 30 22:22:54.056987 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:54.056921    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a6cc21e10bc1e0353ea3b36fe7f807ea2a367e761da2d139b8e0f3b8009c1f0"} err="failed to get container status \"6a6cc21e10bc1e0353ea3b36fe7f807ea2a367e761da2d139b8e0f3b8009c1f0\": rpc error: code = NotFound desc = could not find container \"6a6cc21e10bc1e0353ea3b36fe7f807ea2a367e761da2d139b8e0f3b8009c1f0\": container with ID starting with 6a6cc21e10bc1e0353ea3b36fe7f807ea2a367e761da2d139b8e0f3b8009c1f0 not found: ID does not exist"
Oct 30 22:22:54.056987 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:54.056945    5061 scope.go:122] "RemoveContainer" containerID="28d948b4308d1a2ef8191c64f52379ad5384f4efea11fb8817b380e34d76df2d"
Oct 30 22:22:54.057256 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:22:54.057235    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28d948b4308d1a2ef8191c64f52379ad5384f4efea11fb8817b380e34d76df2d\": container with ID starting with 28d948b4308d1a2ef8191c64f52379ad5384f4efea11fb8817b380e34d76df2d not found: ID does not exist" containerID="28d948b4308d1a2ef8191c64f52379ad5384f4efea11fb8817b380e34d76df2d"
Oct 30 22:22:54.057327 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:54.057260    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28d948b4308d1a2ef8191c64f52379ad5384f4efea11fb8817b380e34d76df2d"} err="failed to get container status \"28d948b4308d1a2ef8191c64f52379ad5384f4efea11fb8817b380e34d76df2d\": rpc error: code = NotFound desc = could not find container \"28d948b4308d1a2ef8191c64f52379ad5384f4efea11fb8817b380e34d76df2d\": container with ID starting with 28d948b4308d1a2ef8191c64f52379ad5384f4efea11fb8817b380e34d76df2d not found: ID does not exist"
Oct 30 22:22:54.057327 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:54.057275    5061 scope.go:122] "RemoveContainer" containerID="69650f186337a304c285b07f94defc757c4f41455ed19bfc01752e056d037b6c"
Oct 30 22:22:54.057832 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:54.057810    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/598138a5-2e71-4727-9332-3c604144f0ef-kube-api-access-6zkwn" pod "598138a5-2e71-4727-9332-3c604144f0ef" (UID: "598138a5-2e71-4727-9332-3c604144f0ef"). InnerVolumeSpecName "kube-api-access-6zkwn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:22:54.069419 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:54.069378    5061 scope.go:122] "RemoveContainer" containerID="69650f186337a304c285b07f94defc757c4f41455ed19bfc01752e056d037b6c"
Oct 30 22:22:54.069728 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:22:54.069691    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69650f186337a304c285b07f94defc757c4f41455ed19bfc01752e056d037b6c\": container with ID starting with 69650f186337a304c285b07f94defc757c4f41455ed19bfc01752e056d037b6c not found: ID does not exist" containerID="69650f186337a304c285b07f94defc757c4f41455ed19bfc01752e056d037b6c"
Oct 30 22:22:54.069814 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:54.069738    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69650f186337a304c285b07f94defc757c4f41455ed19bfc01752e056d037b6c"} err="failed to get container status \"69650f186337a304c285b07f94defc757c4f41455ed19bfc01752e056d037b6c\": rpc error: code = NotFound desc = could not find container \"69650f186337a304c285b07f94defc757c4f41455ed19bfc01752e056d037b6c\": container with ID starting with 69650f186337a304c285b07f94defc757c4f41455ed19bfc01752e056d037b6c not found: ID does not exist"
Oct 30 22:22:54.069814 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:54.069760    5061 scope.go:122] "RemoveContainer" containerID="9cb0b7e5f37d1b429283bcdb104977b0c36564f7cf0cc9f930c4493a4c93fe50"
Oct 30 22:22:54.083330 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:54.083312    5061 scope.go:122] "RemoveContainer" containerID="9cb0b7e5f37d1b429283bcdb104977b0c36564f7cf0cc9f930c4493a4c93fe50"
Oct 30 22:22:54.083614 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:22:54.083591    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9cb0b7e5f37d1b429283bcdb104977b0c36564f7cf0cc9f930c4493a4c93fe50\": container with ID starting with 9cb0b7e5f37d1b429283bcdb104977b0c36564f7cf0cc9f930c4493a4c93fe50 not found: ID does not exist" containerID="9cb0b7e5f37d1b429283bcdb104977b0c36564f7cf0cc9f930c4493a4c93fe50"
Oct 30 22:22:54.083692 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:54.083624    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cb0b7e5f37d1b429283bcdb104977b0c36564f7cf0cc9f930c4493a4c93fe50"} err="failed to get container status \"9cb0b7e5f37d1b429283bcdb104977b0c36564f7cf0cc9f930c4493a4c93fe50\": rpc error: code = NotFound desc = could not find container \"9cb0b7e5f37d1b429283bcdb104977b0c36564f7cf0cc9f930c4493a4c93fe50\": container with ID starting with 9cb0b7e5f37d1b429283bcdb104977b0c36564f7cf0cc9f930c4493a4c93fe50 not found: ID does not exist"
Oct 30 22:22:54.083692 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:54.083644    5061 scope.go:122] "RemoveContainer" containerID="b38d0702c46a0f99ecacb9cbe7ed469a5bd18364eb5845fc32519c06825cc329"
Oct 30 22:22:54.095695 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:54.094924    5061 scope.go:122] "RemoveContainer" containerID="b38d0702c46a0f99ecacb9cbe7ed469a5bd18364eb5845fc32519c06825cc329"
Oct 30 22:22:54.095999 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:22:54.095979    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b38d0702c46a0f99ecacb9cbe7ed469a5bd18364eb5845fc32519c06825cc329\": container with ID starting with b38d0702c46a0f99ecacb9cbe7ed469a5bd18364eb5845fc32519c06825cc329 not found: ID does not exist" containerID="b38d0702c46a0f99ecacb9cbe7ed469a5bd18364eb5845fc32519c06825cc329"
Oct 30 22:22:54.096069 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:54.096020    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b38d0702c46a0f99ecacb9cbe7ed469a5bd18364eb5845fc32519c06825cc329"} err="failed to get container status \"b38d0702c46a0f99ecacb9cbe7ed469a5bd18364eb5845fc32519c06825cc329\": rpc error: code = NotFound desc = could not find container \"b38d0702c46a0f99ecacb9cbe7ed469a5bd18364eb5845fc32519c06825cc329\": container with ID starting with b38d0702c46a0f99ecacb9cbe7ed469a5bd18364eb5845fc32519c06825cc329 not found: ID does not exist"
Oct 30 22:22:54.150191 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:54.150156    5061 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/598138a5-2e71-4727-9332-3c604144f0ef-istiod-ca-cert\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:54.150191 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:54.150193    5061 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/598138a5-2e71-4727-9332-3c604144f0ef-istio-envoy\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:54.150392 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:54.150208    5061 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/598138a5-2e71-4727-9332-3c604144f0ef-workload-socket\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:54.150392 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:54.150224    5061 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/598138a5-2e71-4727-9332-3c604144f0ef-credential-socket\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:54.150392 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:54.150237    5061 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/598138a5-2e71-4727-9332-3c604144f0ef-istio-podinfo\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:54.150392 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:54.150251    5061 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/598138a5-2e71-4727-9332-3c604144f0ef-istio-data\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:54.150392 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:54.150264    5061 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/598138a5-2e71-4727-9332-3c604144f0ef-istio-token\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:54.150392 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:54.150277    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zkwn\" (UniqueName: \"kubernetes.io/projected/598138a5-2e71-4727-9332-3c604144f0ef-kube-api-access-6zkwn\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:54.367248 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:54.367215    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-9r5c8"]
Oct 30 22:22:54.373199 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:54.373169    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-9r5c8"]
Oct 30 22:22:54.532247 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:54.532206    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83adee7e-3585-4d0d-b1b2-d4b5d3be0588" path="/var/lib/kubelet/pods/83adee7e-3585-4d0d-b1b2-d4b5d3be0588/volumes"
Oct 30 22:22:54.604885 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:54.604805    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-nv6kl"]
Oct 30 22:22:54.616164 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:54.616138    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-nv6kl"]
Oct 30 22:22:54.667987 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:22:54.667956    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 22:22:54.722195 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:54.722167    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-k5knf"]
Oct 30 22:22:54.735264 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:54.735234    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-k5knf"]
Oct 30 22:22:54.843640 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:54.843603    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-649qh"]
Oct 30 22:22:54.846944 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:54.846918    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-649qh"]
Oct 30 22:22:55.095759 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:55.095731    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-5be0d1-6989644d59-xk2lk"]
Oct 30 22:22:56.529967 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:56.529925    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3489262b-293e-4dba-8bb2-efa4333ab66d" path="/var/lib/kubelet/pods/3489262b-293e-4dba-8bb2-efa4333ab66d/volumes"
Oct 30 22:22:56.530639 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:56.530617    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="598138a5-2e71-4727-9332-3c604144f0ef" path="/var/lib/kubelet/pods/598138a5-2e71-4727-9332-3c604144f0ef/volumes"
Oct 30 22:22:56.531125 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:56.531104    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="732b6214-aa8c-45fc-bd06-c0f75a267960" path="/var/lib/kubelet/pods/732b6214-aa8c-45fc-bd06-c0f75a267960/volumes"
Oct 30 22:22:59.043394 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:59.043362    5061 generic.go:356] "Generic (PLEG): container finished" podID="8cf03377-68fe-44d3-a000-3a4893ef6b8a" containerID="3b1e4f5beb386b9fd37021e9ec9fb3dc409435a41568715df942b5d8c5e1b869" exitCode=0
Oct 30 22:22:59.043863 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:59.043442    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-84a3cf-openshift-default-6b47cf679-n6j8d" event={"ID":"8cf03377-68fe-44d3-a000-3a4893ef6b8a","Type":"ContainerDied","Data":"3b1e4f5beb386b9fd37021e9ec9fb3dc409435a41568715df942b5d8c5e1b869"}
Oct 30 22:22:59.043863 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:59.043806    5061 scope.go:122] "RemoveContainer" containerID="3b1e4f5beb386b9fd37021e9ec9fb3dc409435a41568715df942b5d8c5e1b869"
Oct 30 22:22:59.045349 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:59.045326    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:22:59.045421 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:59.045389    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:22:59.045421 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:22:59.045418    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:23:00.052578 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:00.052518    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-84a3cf-openshift-default-6b47cf679-n6j8d" event={"ID":"8cf03377-68fe-44d3-a000-3a4893ef6b8a","Type":"ContainerStarted","Data":"e5be667ce8b3fac749e9c6931a15edeb4f2b971358bd3cc24855c5f01ded15d9"}
Oct 30 22:23:00.097021 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:00.096966    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-5be0d1-6989644d59-xk2lk" podUID="36373ff4-1396-4b98-8f21-8e261b2e0268" containerName="payload-pre-processing" containerID="cri-o://415debea16a93bcde53c7d946768352b6cd0f0878606f3e6a4a7c1f530099f2f" gracePeriod=25
Oct 30 22:23:00.605355 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:00.605323    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-84a3cf-openshift-default-6b47cf679-n6j8d"
Oct 30 22:23:00.605535 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:00.605446    5061 patch_prober.go:29] interesting pod/e2e-inf-b-84a3cf-openshift-default-6b47cf679-n6j8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body=
Oct 30 22:23:00.605535 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:00.605502    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-84a3cf-openshift-default-6b47cf679-n6j8d" podUID="8cf03377-68fe-44d3-a000-3a4893ef6b8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused"
Oct 30 22:23:01.605366 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:01.605328    5061 patch_prober.go:29] interesting pod/e2e-inf-b-84a3cf-openshift-default-6b47cf679-n6j8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body=
Oct 30 22:23:01.605781 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:01.605389    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-84a3cf-openshift-default-6b47cf679-n6j8d" podUID="8cf03377-68fe-44d3-a000-3a4893ef6b8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused"
Oct 30 22:23:02.434811 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:02.434771    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-5be0d1/distinct-w2-5be0d1-kserve-6b85698c8-sxn9k" podUID="7a2dbfc5-908f-41d1-baff-074c56d2cf16" containerName="main" containerID="cri-o://eec9588e46967a83b7f37b412b19ec567120263bae9533e119ae2b46151c015b" gracePeriod=45
Oct 30 22:23:02.448979 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:02.448945    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-5be0d1/premium-w2-5be0d1-kserve-759b6fb995-c6gp4" podUID="933f2470-68de-4768-a05b-6c454f16f5f2" containerName="main" containerID="cri-o://f511e810551d2e435d6289dbdbb835de89b6cf797170c3f5615ae6351f28459c" gracePeriod=45
Oct 30 22:23:02.605799 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:02.605761    5061 patch_prober.go:29] interesting pod/e2e-inf-b-84a3cf-openshift-default-6b47cf679-n6j8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body=
Oct 30 22:23:02.606155 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:02.605837    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-84a3cf-openshift-default-6b47cf679-n6j8d" podUID="8cf03377-68fe-44d3-a000-3a4893ef6b8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused"
Oct 30 22:23:02.763891 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:02.763849    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-5be0d1/distinct-w2-5be0d1-kserve-6b85698c8-sxn9k"
Oct 30 22:23:02.796686 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:02.796665    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-5be0d1/premium-w2-5be0d1-kserve-759b6fb995-c6gp4"
Oct 30 22:23:02.872888 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:02.872851    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/933f2470-68de-4768-a05b-6c454f16f5f2-kube-api-access-grw7m\" (UniqueName: \"kubernetes.io/projected/933f2470-68de-4768-a05b-6c454f16f5f2-kube-api-access-grw7m\") pod \"933f2470-68de-4768-a05b-6c454f16f5f2\" (UID: \"933f2470-68de-4768-a05b-6c454f16f5f2\") "
Oct 30 22:23:02.873080 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:02.872918    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a2dbfc5-908f-41d1-baff-074c56d2cf16-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a2dbfc5-908f-41d1-baff-074c56d2cf16-dshm\") pod \"7a2dbfc5-908f-41d1-baff-074c56d2cf16\" (UID: \"7a2dbfc5-908f-41d1-baff-074c56d2cf16\") "
Oct 30 22:23:02.873080 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:02.872951    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/933f2470-68de-4768-a05b-6c454f16f5f2-home\" (UniqueName: \"kubernetes.io/empty-dir/933f2470-68de-4768-a05b-6c454f16f5f2-home\") pod \"933f2470-68de-4768-a05b-6c454f16f5f2\" (UID: \"933f2470-68de-4768-a05b-6c454f16f5f2\") "
Oct 30 22:23:02.873080 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:02.873003    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a2dbfc5-908f-41d1-baff-074c56d2cf16-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a2dbfc5-908f-41d1-baff-074c56d2cf16-model-cache\") pod \"7a2dbfc5-908f-41d1-baff-074c56d2cf16\" (UID: \"7a2dbfc5-908f-41d1-baff-074c56d2cf16\") "
Oct 30 22:23:02.873080 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:02.873040    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/933f2470-68de-4768-a05b-6c454f16f5f2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/933f2470-68de-4768-a05b-6c454f16f5f2-model-cache\") pod \"933f2470-68de-4768-a05b-6c454f16f5f2\" (UID: \"933f2470-68de-4768-a05b-6c454f16f5f2\") "
Oct 30 22:23:02.873080 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:02.873066    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/933f2470-68de-4768-a05b-6c454f16f5f2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/933f2470-68de-4768-a05b-6c454f16f5f2-tmp-dir\") pod \"933f2470-68de-4768-a05b-6c454f16f5f2\" (UID: \"933f2470-68de-4768-a05b-6c454f16f5f2\") "
Oct 30 22:23:02.873377 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:02.873089    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a2dbfc5-908f-41d1-baff-074c56d2cf16-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a2dbfc5-908f-41d1-baff-074c56d2cf16-tmp-dir\") pod \"7a2dbfc5-908f-41d1-baff-074c56d2cf16\" (UID: \"7a2dbfc5-908f-41d1-baff-074c56d2cf16\") "
Oct 30 22:23:02.873377 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:02.873193    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/933f2470-68de-4768-a05b-6c454f16f5f2-tls-certs\" (UniqueName: \"kubernetes.io/secret/933f2470-68de-4768-a05b-6c454f16f5f2-tls-certs\") pod \"933f2470-68de-4768-a05b-6c454f16f5f2\" (UID: \"933f2470-68de-4768-a05b-6c454f16f5f2\") "
Oct 30 22:23:02.873377 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:02.873257    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/933f2470-68de-4768-a05b-6c454f16f5f2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/933f2470-68de-4768-a05b-6c454f16f5f2-dshm\") pod \"933f2470-68de-4768-a05b-6c454f16f5f2\" (UID: \"933f2470-68de-4768-a05b-6c454f16f5f2\") "
Oct 30 22:23:02.873377 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:02.873314    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a2dbfc5-908f-41d1-baff-074c56d2cf16-model-cache" pod "7a2dbfc5-908f-41d1-baff-074c56d2cf16" (UID: "7a2dbfc5-908f-41d1-baff-074c56d2cf16"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:23:02.873377 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:02.873343    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a2dbfc5-908f-41d1-baff-074c56d2cf16-home\" (UniqueName: \"kubernetes.io/empty-dir/7a2dbfc5-908f-41d1-baff-074c56d2cf16-home\") pod \"7a2dbfc5-908f-41d1-baff-074c56d2cf16\" (UID: \"7a2dbfc5-908f-41d1-baff-074c56d2cf16\") "
Oct 30 22:23:02.873669 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:02.873409    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a2dbfc5-908f-41d1-baff-074c56d2cf16-kube-api-access-xv4l8\" (UniqueName: \"kubernetes.io/projected/7a2dbfc5-908f-41d1-baff-074c56d2cf16-kube-api-access-xv4l8\") pod \"7a2dbfc5-908f-41d1-baff-074c56d2cf16\" (UID: \"7a2dbfc5-908f-41d1-baff-074c56d2cf16\") "
Oct 30 22:23:02.873669 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:02.873441    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a2dbfc5-908f-41d1-baff-074c56d2cf16-tls-certs\" (UniqueName: \"kubernetes.io/secret/7a2dbfc5-908f-41d1-baff-074c56d2cf16-tls-certs\") pod \"7a2dbfc5-908f-41d1-baff-074c56d2cf16\" (UID: \"7a2dbfc5-908f-41d1-baff-074c56d2cf16\") "
Oct 30 22:23:02.873787 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:02.873733    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/933f2470-68de-4768-a05b-6c454f16f5f2-home" pod "933f2470-68de-4768-a05b-6c454f16f5f2" (UID: "933f2470-68de-4768-a05b-6c454f16f5f2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:23:02.874099 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:02.874075    5061 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/933f2470-68de-4768-a05b-6c454f16f5f2-home\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:02.874184 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:02.874103    5061 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a2dbfc5-908f-41d1-baff-074c56d2cf16-model-cache\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:02.874458 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:02.874273    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a2dbfc5-908f-41d1-baff-074c56d2cf16-tmp-dir" pod "7a2dbfc5-908f-41d1-baff-074c56d2cf16" (UID: "7a2dbfc5-908f-41d1-baff-074c56d2cf16"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:23:02.874458 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:02.874289    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a2dbfc5-908f-41d1-baff-074c56d2cf16-home" pod "7a2dbfc5-908f-41d1-baff-074c56d2cf16" (UID: "7a2dbfc5-908f-41d1-baff-074c56d2cf16"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:23:02.874640 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:02.874217    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/933f2470-68de-4768-a05b-6c454f16f5f2-tmp-dir" pod "933f2470-68de-4768-a05b-6c454f16f5f2" (UID: "933f2470-68de-4768-a05b-6c454f16f5f2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:23:02.874706 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:02.874655    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/933f2470-68de-4768-a05b-6c454f16f5f2-model-cache" pod "933f2470-68de-4768-a05b-6c454f16f5f2" (UID: "933f2470-68de-4768-a05b-6c454f16f5f2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:23:02.875798 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:02.875769    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/933f2470-68de-4768-a05b-6c454f16f5f2-kube-api-access-grw7m" pod "933f2470-68de-4768-a05b-6c454f16f5f2" (UID: "933f2470-68de-4768-a05b-6c454f16f5f2"). InnerVolumeSpecName "kube-api-access-grw7m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:23:02.876629 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:02.876604    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/933f2470-68de-4768-a05b-6c454f16f5f2-tls-certs" pod "933f2470-68de-4768-a05b-6c454f16f5f2" (UID: "933f2470-68de-4768-a05b-6c454f16f5f2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:23:02.876971 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:02.876946    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a2dbfc5-908f-41d1-baff-074c56d2cf16-dshm" pod "7a2dbfc5-908f-41d1-baff-074c56d2cf16" (UID: "7a2dbfc5-908f-41d1-baff-074c56d2cf16"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:23:02.876971 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:02.876958    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/933f2470-68de-4768-a05b-6c454f16f5f2-dshm" pod "933f2470-68de-4768-a05b-6c454f16f5f2" (UID: "933f2470-68de-4768-a05b-6c454f16f5f2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:23:02.877122 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:02.877099    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a2dbfc5-908f-41d1-baff-074c56d2cf16-kube-api-access-xv4l8" pod "7a2dbfc5-908f-41d1-baff-074c56d2cf16" (UID: "7a2dbfc5-908f-41d1-baff-074c56d2cf16"). InnerVolumeSpecName "kube-api-access-xv4l8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:23:02.877225 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:02.877210    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a2dbfc5-908f-41d1-baff-074c56d2cf16-tls-certs" pod "7a2dbfc5-908f-41d1-baff-074c56d2cf16" (UID: "7a2dbfc5-908f-41d1-baff-074c56d2cf16"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:23:02.975838 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:02.975806    5061 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/933f2470-68de-4768-a05b-6c454f16f5f2-tls-certs\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:02.975838 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:02.975832    5061 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/933f2470-68de-4768-a05b-6c454f16f5f2-dshm\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:02.975838 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:02.975841    5061 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a2dbfc5-908f-41d1-baff-074c56d2cf16-home\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:02.976078 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:02.975849    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xv4l8\" (UniqueName: \"kubernetes.io/projected/7a2dbfc5-908f-41d1-baff-074c56d2cf16-kube-api-access-xv4l8\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:02.976078 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:02.975859    5061 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a2dbfc5-908f-41d1-baff-074c56d2cf16-tls-certs\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:02.976078 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:02.975867    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grw7m\" (UniqueName: \"kubernetes.io/projected/933f2470-68de-4768-a05b-6c454f16f5f2-kube-api-access-grw7m\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:02.976078 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:02.975885    5061 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a2dbfc5-908f-41d1-baff-074c56d2cf16-dshm\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:02.976078 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:02.975893    5061 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/933f2470-68de-4768-a05b-6c454f16f5f2-model-cache\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:02.976078 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:02.975900    5061 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/933f2470-68de-4768-a05b-6c454f16f5f2-tmp-dir\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:02.976078 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:02.975908    5061 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a2dbfc5-908f-41d1-baff-074c56d2cf16-tmp-dir\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:03.077907 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:03.077873    5061 generic.go:356] "Generic (PLEG): container finished" podID="7a2dbfc5-908f-41d1-baff-074c56d2cf16" containerID="eec9588e46967a83b7f37b412b19ec567120263bae9533e119ae2b46151c015b" exitCode=0
Oct 30 22:23:03.078100 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:03.077916    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-5be0d1/distinct-w2-5be0d1-kserve-6b85698c8-sxn9k"
Oct 30 22:23:03.078100 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:03.077962    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-5be0d1/distinct-w2-5be0d1-kserve-6b85698c8-sxn9k" event={"ID":"7a2dbfc5-908f-41d1-baff-074c56d2cf16","Type":"ContainerDied","Data":"eec9588e46967a83b7f37b412b19ec567120263bae9533e119ae2b46151c015b"}
Oct 30 22:23:03.078100 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:03.078027    5061 scope.go:122] "RemoveContainer" containerID="eec9588e46967a83b7f37b412b19ec567120263bae9533e119ae2b46151c015b"
Oct 30 22:23:03.079675 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:03.078920    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-5be0d1/distinct-w2-5be0d1-kserve-6b85698c8-sxn9k" event={"ID":"7a2dbfc5-908f-41d1-baff-074c56d2cf16","Type":"ContainerDied","Data":"3e16833a8d2df226baf3ffea5b9c50c4a1b5be08d2f53f497c7030700ef2d9f8"}
Oct 30 22:23:03.081351 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:03.081219    5061 generic.go:356] "Generic (PLEG): container finished" podID="933f2470-68de-4768-a05b-6c454f16f5f2" containerID="f511e810551d2e435d6289dbdbb835de89b6cf797170c3f5615ae6351f28459c" exitCode=0
Oct 30 22:23:03.081351 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:03.081255    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-5be0d1/premium-w2-5be0d1-kserve-759b6fb995-c6gp4" event={"ID":"933f2470-68de-4768-a05b-6c454f16f5f2","Type":"ContainerDied","Data":"f511e810551d2e435d6289dbdbb835de89b6cf797170c3f5615ae6351f28459c"}
Oct 30 22:23:03.081351 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:03.081313    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-5be0d1/premium-w2-5be0d1-kserve-759b6fb995-c6gp4" event={"ID":"933f2470-68de-4768-a05b-6c454f16f5f2","Type":"ContainerDied","Data":"29e316f76da4cd7ec06b6c1c82720dd9f32b432d9df4776b8225fcb281dc8eee"}
Oct 30 22:23:03.081677 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:03.081520    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-5be0d1/premium-w2-5be0d1-kserve-759b6fb995-c6gp4"
Oct 30 22:23:03.091752 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:03.091732    5061 scope.go:122] "RemoveContainer" containerID="eec9588e46967a83b7f37b412b19ec567120263bae9533e119ae2b46151c015b"
Oct 30 22:23:03.092014 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:23:03.091998    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eec9588e46967a83b7f37b412b19ec567120263bae9533e119ae2b46151c015b\": container with ID starting with eec9588e46967a83b7f37b412b19ec567120263bae9533e119ae2b46151c015b not found: ID does not exist" containerID="eec9588e46967a83b7f37b412b19ec567120263bae9533e119ae2b46151c015b"
Oct 30 22:23:03.092074 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:03.092022    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eec9588e46967a83b7f37b412b19ec567120263bae9533e119ae2b46151c015b"} err="failed to get container status \"eec9588e46967a83b7f37b412b19ec567120263bae9533e119ae2b46151c015b\": rpc error: code = NotFound desc = could not find container \"eec9588e46967a83b7f37b412b19ec567120263bae9533e119ae2b46151c015b\": container with ID starting with eec9588e46967a83b7f37b412b19ec567120263bae9533e119ae2b46151c015b not found: ID does not exist"
Oct 30 22:23:03.092074 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:03.092036    5061 scope.go:122] "RemoveContainer" containerID="f511e810551d2e435d6289dbdbb835de89b6cf797170c3f5615ae6351f28459c"
Oct 30 22:23:03.101295 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:03.101275    5061 scope.go:122] "RemoveContainer" containerID="f511e810551d2e435d6289dbdbb835de89b6cf797170c3f5615ae6351f28459c"
Oct 30 22:23:03.101585 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:23:03.101567    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f511e810551d2e435d6289dbdbb835de89b6cf797170c3f5615ae6351f28459c\": container with ID starting with f511e810551d2e435d6289dbdbb835de89b6cf797170c3f5615ae6351f28459c not found: ID does not exist" containerID="f511e810551d2e435d6289dbdbb835de89b6cf797170c3f5615ae6351f28459c"
Oct 30 22:23:03.101646 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:03.101590    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f511e810551d2e435d6289dbdbb835de89b6cf797170c3f5615ae6351f28459c"} err="failed to get container status \"f511e810551d2e435d6289dbdbb835de89b6cf797170c3f5615ae6351f28459c\": rpc error: code = NotFound desc = could not find container \"f511e810551d2e435d6289dbdbb835de89b6cf797170c3f5615ae6351f28459c\": container with ID starting with f511e810551d2e435d6289dbdbb835de89b6cf797170c3f5615ae6351f28459c not found: ID does not exist"
Oct 30 22:23:03.240108 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:03.240079    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-5be0d1/premium-w2-5be0d1-kserve-759b6fb995-c6gp4"]
Oct 30 22:23:03.251355 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:03.251327    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-5be0d1/premium-w2-5be0d1-kserve-759b6fb995-c6gp4"]
Oct 30 22:23:03.373168 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:03.373057    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-5be0d1/distinct-w2-5be0d1-kserve-6b85698c8-sxn9k"]
Oct 30 22:23:03.374242 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:03.374213    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-5be0d1/distinct-w2-5be0d1-kserve-6b85698c8-sxn9k"]
Oct 30 22:23:03.604919 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:03.604886    5061 patch_prober.go:29] interesting pod/e2e-inf-b-84a3cf-openshift-default-6b47cf679-n6j8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body=
Oct 30 22:23:03.605086 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:03.604946    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-84a3cf-openshift-default-6b47cf679-n6j8d" podUID="8cf03377-68fe-44d3-a000-3a4893ef6b8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused"
Oct 30 22:23:04.562477 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:04.562437    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a2dbfc5-908f-41d1-baff-074c56d2cf16" path="/var/lib/kubelet/pods/7a2dbfc5-908f-41d1-baff-074c56d2cf16/volumes"
Oct 30 22:23:04.563401 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:04.563368    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="933f2470-68de-4768-a05b-6c454f16f5f2" path="/var/lib/kubelet/pods/933f2470-68de-4768-a05b-6c454f16f5f2/volumes"
Oct 30 22:23:04.604835 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:04.604800    5061 patch_prober.go:29] interesting pod/e2e-inf-b-84a3cf-openshift-default-6b47cf679-n6j8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body=
Oct 30 22:23:04.604989 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:04.604865    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-84a3cf-openshift-default-6b47cf679-n6j8d" podUID="8cf03377-68fe-44d3-a000-3a4893ef6b8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused"
Oct 30 22:23:05.610565 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:05.610515    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-84a3cf-openshift-default-6b47cf679-n6j8d"
Oct 30 22:23:05.611049 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:05.611029    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-84a3cf-openshift-default-6b47cf679-n6j8d"
Oct 30 22:23:05.611787 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:05.611764    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-84a3cf-openshift-default-6b47cf679-n6j8d"
Oct 30 22:23:06.653874 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:06.653841    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0ce563-vqwjg"]
Oct 30 22:23:06.659578 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:06.659552    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0ce563-vqwjg"]
Oct 30 22:23:08.529281 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:08.529243    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6954b8cd-7365-435f-b281-d5339ac3df13" path="/var/lib/kubelet/pods/6954b8cd-7365-435f-b281-d5339ac3df13/volumes"
Oct 30 22:23:09.844826 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:09.844790    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-5be0d1-6989644d59-xk2lk" podUID="36373ff4-1396-4b98-8f21-8e261b2e0268" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:23:10.982752 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:10.982719    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2d9b5eb-ws4sd"]
Oct 30 22:23:10.983859 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:10.983834    5061 cpu_manager.go:414] "containerMap: removing container" podUID="598138a5-2e71-4727-9332-3c604144f0ef" containerName="istio-proxy"
Oct 30 22:23:10.983968 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:10.983878    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="598138a5-2e71-4727-9332-3c604144f0ef" containerName="istio-proxy"
Oct 30 22:23:10.983968 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:10.983901    5061 cpu_manager.go:414] "containerMap: removing container" podUID="7a2dbfc5-908f-41d1-baff-074c56d2cf16" containerName="main"
Oct 30 22:23:10.983968 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:10.983909    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a2dbfc5-908f-41d1-baff-074c56d2cf16" containerName="main"
Oct 30 22:23:10.983968 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:10.983923    5061 cpu_manager.go:414] "containerMap: removing container" podUID="732b6214-aa8c-45fc-bd06-c0f75a267960" containerName="istio-proxy"
Oct 30 22:23:10.983968 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:10.983931    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="732b6214-aa8c-45fc-bd06-c0f75a267960" containerName="istio-proxy"
Oct 30 22:23:10.984149 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:10.983976    5061 cpu_manager.go:414] "containerMap: removing container" podUID="83adee7e-3585-4d0d-b1b2-d4b5d3be0588" containerName="istio-proxy"
Oct 30 22:23:10.984149 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:10.983984    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83adee7e-3585-4d0d-b1b2-d4b5d3be0588" containerName="istio-proxy"
Oct 30 22:23:10.984149 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:10.983994    5061 cpu_manager.go:414] "containerMap: removing container" podUID="83adee7e-3585-4d0d-b1b2-d4b5d3be0588" containerName="istio-proxy"
Oct 30 22:23:10.984149 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:10.984001    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83adee7e-3585-4d0d-b1b2-d4b5d3be0588" containerName="istio-proxy"
Oct 30 22:23:10.984149 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:10.984017    5061 cpu_manager.go:414] "containerMap: removing container" podUID="3489262b-293e-4dba-8bb2-efa4333ab66d" containerName="istio-proxy"
Oct 30 22:23:10.984149 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:10.984024    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3489262b-293e-4dba-8bb2-efa4333ab66d" containerName="istio-proxy"
Oct 30 22:23:10.984149 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:10.984036    5061 cpu_manager.go:414] "containerMap: removing container" podUID="efe7c063-fb0e-4679-8a7c-13f2a0abaa74" containerName="main"
Oct 30 22:23:10.984149 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:10.984042    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efe7c063-fb0e-4679-8a7c-13f2a0abaa74" containerName="main"
Oct 30 22:23:10.984149 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:10.984055    5061 cpu_manager.go:414] "containerMap: removing container" podUID="933f2470-68de-4768-a05b-6c454f16f5f2" containerName="main"
Oct 30 22:23:10.984149 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:10.984061    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="933f2470-68de-4768-a05b-6c454f16f5f2" containerName="main"
Oct 30 22:23:10.984149 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:10.984072    5061 cpu_manager.go:414] "containerMap: removing container" podUID="17eb78ae-158c-49f2-8ff2-b7f1a7918a82" containerName="revoke-keys"
Oct 30 22:23:10.984149 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:10.984078    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17eb78ae-158c-49f2-8ff2-b7f1a7918a82" containerName="revoke-keys"
Oct 30 22:23:10.984499 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:10.984198    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="732b6214-aa8c-45fc-bd06-c0f75a267960" containerName="istio-proxy"
Oct 30 22:23:10.984499 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:10.984213    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="83adee7e-3585-4d0d-b1b2-d4b5d3be0588" containerName="istio-proxy"
Oct 30 22:23:10.984499 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:10.984221    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="83adee7e-3585-4d0d-b1b2-d4b5d3be0588" containerName="istio-proxy"
Oct 30 22:23:10.984499 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:10.984228    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="3489262b-293e-4dba-8bb2-efa4333ab66d" containerName="istio-proxy"
Oct 30 22:23:10.984499 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:10.984234    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="efe7c063-fb0e-4679-8a7c-13f2a0abaa74" containerName="main"
Oct 30 22:23:10.984499 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:10.984239    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="933f2470-68de-4768-a05b-6c454f16f5f2" containerName="main"
Oct 30 22:23:10.984499 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:10.984245    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="17eb78ae-158c-49f2-8ff2-b7f1a7918a82" containerName="revoke-keys"
Oct 30 22:23:10.984499 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:10.984252    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="598138a5-2e71-4727-9332-3c604144f0ef" containerName="istio-proxy"
Oct 30 22:23:10.984499 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:10.984260    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a2dbfc5-908f-41d1-baff-074c56d2cf16" containerName="main"
Oct 30 22:23:10.989252 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:10.989227    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2d9b5eb-ws4sd"
Oct 30 22:23:11.083609 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:11.083552    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa64ef6b-8386-4f0b-b741-b77e4b5d4865-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f2d9b5eb-ws4sd\" (UID: \"fa64ef6b-8386-4f0b-b741-b77e4b5d4865\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2d9b5eb-ws4sd"
Oct 30 22:23:11.083784 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:11.083661    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d728h\" (UniqueName: \"kubernetes.io/projected/fa64ef6b-8386-4f0b-b741-b77e4b5d4865-kube-api-access-d728h\") pod \"maas-api-revoke-sub-f2d9b5eb-ws4sd\" (UID: \"fa64ef6b-8386-4f0b-b741-b77e4b5d4865\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2d9b5eb-ws4sd"
Oct 30 22:23:11.083784 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:11.083708    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa64ef6b-8386-4f0b-b741-b77e4b5d4865-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f2d9b5eb-ws4sd\" (UID: \"fa64ef6b-8386-4f0b-b741-b77e4b5d4865\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2d9b5eb-ws4sd"
Oct 30 22:23:11.088125 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:11.088099    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:23:11.088265 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:11.088238    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gdmfg\""
Oct 30 22:23:11.093600 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:11.093576    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2d9b5eb-ws4sd"]
Oct 30 22:23:11.185094 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:11.185069    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa64ef6b-8386-4f0b-b741-b77e4b5d4865-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f2d9b5eb-ws4sd\" (UID: \"fa64ef6b-8386-4f0b-b741-b77e4b5d4865\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2d9b5eb-ws4sd"
Oct 30 22:23:11.185234 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:11.185112    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d728h\" (UniqueName: \"kubernetes.io/projected/fa64ef6b-8386-4f0b-b741-b77e4b5d4865-kube-api-access-d728h\") pod \"maas-api-revoke-sub-f2d9b5eb-ws4sd\" (UID: \"fa64ef6b-8386-4f0b-b741-b77e4b5d4865\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2d9b5eb-ws4sd"
Oct 30 22:23:11.185234 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:11.185148    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa64ef6b-8386-4f0b-b741-b77e4b5d4865-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f2d9b5eb-ws4sd\" (UID: \"fa64ef6b-8386-4f0b-b741-b77e4b5d4865\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2d9b5eb-ws4sd"
Oct 30 22:23:11.185850 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:11.185827    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa64ef6b-8386-4f0b-b741-b77e4b5d4865-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f2d9b5eb-ws4sd\" (UID: \"fa64ef6b-8386-4f0b-b741-b77e4b5d4865\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2d9b5eb-ws4sd"
Oct 30 22:23:11.187487 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:11.187467    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa64ef6b-8386-4f0b-b741-b77e4b5d4865-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f2d9b5eb-ws4sd\" (UID: \"fa64ef6b-8386-4f0b-b741-b77e4b5d4865\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2d9b5eb-ws4sd"
Oct 30 22:23:11.220910 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:11.220876    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f29a2d08-qzn4l"]
Oct 30 22:23:11.225978 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:11.225961    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f29a2d08-qzn4l"
Oct 30 22:23:11.239959 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:11.239932    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d728h\" (UniqueName: \"kubernetes.io/projected/fa64ef6b-8386-4f0b-b741-b77e4b5d4865-kube-api-access-d728h\") pod \"maas-api-revoke-sub-f2d9b5eb-ws4sd\" (UID: \"fa64ef6b-8386-4f0b-b741-b77e4b5d4865\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2d9b5eb-ws4sd"
Oct 30 22:23:11.282717 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:11.282695    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f29a2d08-qzn4l"]
Oct 30 22:23:11.286004 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:11.285981    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73bba646-34ed-4c37-8cdb-df7b4ea77e7f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f29a2d08-qzn4l\" (UID: \"73bba646-34ed-4c37-8cdb-df7b4ea77e7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f29a2d08-qzn4l"
Oct 30 22:23:11.286097 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:11.286031    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bx8qn\" (UniqueName: \"kubernetes.io/projected/73bba646-34ed-4c37-8cdb-df7b4ea77e7f-kube-api-access-bx8qn\") pod \"maas-api-revoke-sub-f29a2d08-qzn4l\" (UID: \"73bba646-34ed-4c37-8cdb-df7b4ea77e7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f29a2d08-qzn4l"
Oct 30 22:23:11.286171 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:11.286140    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73bba646-34ed-4c37-8cdb-df7b4ea77e7f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f29a2d08-qzn4l\" (UID: \"73bba646-34ed-4c37-8cdb-df7b4ea77e7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f29a2d08-qzn4l"
Oct 30 22:23:11.303625 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:11.303600    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2d9b5eb-ws4sd"
Oct 30 22:23:11.387388 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:11.387354    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73bba646-34ed-4c37-8cdb-df7b4ea77e7f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f29a2d08-qzn4l\" (UID: \"73bba646-34ed-4c37-8cdb-df7b4ea77e7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f29a2d08-qzn4l"
Oct 30 22:23:11.387584 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:11.387413    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bx8qn\" (UniqueName: \"kubernetes.io/projected/73bba646-34ed-4c37-8cdb-df7b4ea77e7f-kube-api-access-bx8qn\") pod \"maas-api-revoke-sub-f29a2d08-qzn4l\" (UID: \"73bba646-34ed-4c37-8cdb-df7b4ea77e7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f29a2d08-qzn4l"
Oct 30 22:23:11.387584 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:11.387468    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73bba646-34ed-4c37-8cdb-df7b4ea77e7f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f29a2d08-qzn4l\" (UID: \"73bba646-34ed-4c37-8cdb-df7b4ea77e7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f29a2d08-qzn4l"
Oct 30 22:23:11.388383 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:11.388352    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73bba646-34ed-4c37-8cdb-df7b4ea77e7f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f29a2d08-qzn4l\" (UID: \"73bba646-34ed-4c37-8cdb-df7b4ea77e7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f29a2d08-qzn4l"
Oct 30 22:23:11.390586 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:11.390563    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73bba646-34ed-4c37-8cdb-df7b4ea77e7f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f29a2d08-qzn4l\" (UID: \"73bba646-34ed-4c37-8cdb-df7b4ea77e7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f29a2d08-qzn4l"
Oct 30 22:23:11.442257 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:11.442223    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bx8qn\" (UniqueName: \"kubernetes.io/projected/73bba646-34ed-4c37-8cdb-df7b4ea77e7f-kube-api-access-bx8qn\") pod \"maas-api-revoke-sub-f29a2d08-qzn4l\" (UID: \"73bba646-34ed-4c37-8cdb-df7b4ea77e7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f29a2d08-qzn4l"
Oct 30 22:23:11.539809 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:11.539726    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f29a2d08-qzn4l"
Oct 30 22:23:11.676776 ip-10-0-130-60 kubenswrapper[5061]: W1030 22:23:11.676745    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa64ef6b_8386_4f0b_b741_b77e4b5d4865.slice/crio-f65d5663f1e8534da40ce0a41715b8fa3ae47716356cb6ba95c9de3805b7a361 WatchSource:0}: Error finding container f65d5663f1e8534da40ce0a41715b8fa3ae47716356cb6ba95c9de3805b7a361: Status 404 returned error can't find the container with id f65d5663f1e8534da40ce0a41715b8fa3ae47716356cb6ba95c9de3805b7a361
Oct 30 22:23:11.676926 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:11.676810    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2d9b5eb-ws4sd"]
Oct 30 22:23:11.911664 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:11.911635    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f29a2d08-qzn4l"]
Oct 30 22:23:11.912331 ip-10-0-130-60 kubenswrapper[5061]: W1030 22:23:11.912307    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73bba646_34ed_4c37_8cdb_df7b4ea77e7f.slice/crio-83a56d80830e648a8c3e88c65f0505591d86335142091cbe20fec63b7b4992cb WatchSource:0}: Error finding container 83a56d80830e648a8c3e88c65f0505591d86335142091cbe20fec63b7b4992cb: Status 404 returned error can't find the container with id 83a56d80830e648a8c3e88c65f0505591d86335142091cbe20fec63b7b4992cb
Oct 30 22:23:12.149749 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:12.149716    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f29a2d08-qzn4l" event={"ID":"73bba646-34ed-4c37-8cdb-df7b4ea77e7f","Type":"ContainerStarted","Data":"83a56d80830e648a8c3e88c65f0505591d86335142091cbe20fec63b7b4992cb"}
Oct 30 22:23:12.151616 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:12.151590    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2d9b5eb-ws4sd" event={"ID":"fa64ef6b-8386-4f0b-b741-b77e4b5d4865","Type":"ContainerStarted","Data":"4afdfbc94d6c42f46da2ee0a8da31e7f99e48a3267eb9491dccbcb90a0ef160a"}
Oct 30 22:23:12.151716 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:12.151635    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2d9b5eb-ws4sd" event={"ID":"fa64ef6b-8386-4f0b-b741-b77e4b5d4865","Type":"ContainerStarted","Data":"f65d5663f1e8534da40ce0a41715b8fa3ae47716356cb6ba95c9de3805b7a361"}
Oct 30 22:23:12.266738 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:12.263656    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2d9b5eb-ws4sd" podStartSLOduration=2.263637894 podStartE2EDuration="2.263637894s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:23:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:23:12.261615155 +0000 UTC m=+2445.874384915" watchObservedRunningTime="2026-10-30 22:23:12.263637894 +0000 UTC m=+2445.876407652"
Oct 30 22:23:13.160947 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:13.160913    5061 generic.go:356] "Generic (PLEG): container finished" podID="fa64ef6b-8386-4f0b-b741-b77e4b5d4865" containerID="4afdfbc94d6c42f46da2ee0a8da31e7f99e48a3267eb9491dccbcb90a0ef160a" exitCode=0
Oct 30 22:23:13.161437 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:13.160996    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2d9b5eb-ws4sd" event={"ID":"fa64ef6b-8386-4f0b-b741-b77e4b5d4865","Type":"ContainerDied","Data":"4afdfbc94d6c42f46da2ee0a8da31e7f99e48a3267eb9491dccbcb90a0ef160a"}
Oct 30 22:23:13.162955 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:13.162931    5061 generic.go:356] "Generic (PLEG): container finished" podID="73bba646-34ed-4c37-8cdb-df7b4ea77e7f" containerID="ec31bf627ac29bb006d6ff6c1104005e6058a4567b86dfb7bf7eed885c05f73a" exitCode=0
Oct 30 22:23:13.163070 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:13.162980    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f29a2d08-qzn4l" event={"ID":"73bba646-34ed-4c37-8cdb-df7b4ea77e7f","Type":"ContainerDied","Data":"ec31bf627ac29bb006d6ff6c1104005e6058a4567b86dfb7bf7eed885c05f73a"}
Oct 30 22:23:14.363054 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:14.363029    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f29a2d08-qzn4l"
Oct 30 22:23:14.366801 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:14.366779    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2d9b5eb-ws4sd"
Oct 30 22:23:14.423940 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:14.423865    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73bba646-34ed-4c37-8cdb-df7b4ea77e7f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73bba646-34ed-4c37-8cdb-df7b4ea77e7f-maas-api-xks-ca\") pod \"73bba646-34ed-4c37-8cdb-df7b4ea77e7f\" (UID: \"73bba646-34ed-4c37-8cdb-df7b4ea77e7f\") "
Oct 30 22:23:14.424078 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:14.423937    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73bba646-34ed-4c37-8cdb-df7b4ea77e7f-kube-api-access-bx8qn\" (UniqueName: \"kubernetes.io/projected/73bba646-34ed-4c37-8cdb-df7b4ea77e7f-kube-api-access-bx8qn\") pod \"73bba646-34ed-4c37-8cdb-df7b4ea77e7f\" (UID: \"73bba646-34ed-4c37-8cdb-df7b4ea77e7f\") "
Oct 30 22:23:14.424078 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:14.424022    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73bba646-34ed-4c37-8cdb-df7b4ea77e7f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73bba646-34ed-4c37-8cdb-df7b4ea77e7f-maas-api-openshift-ca\") pod \"73bba646-34ed-4c37-8cdb-df7b4ea77e7f\" (UID: \"73bba646-34ed-4c37-8cdb-df7b4ea77e7f\") "
Oct 30 22:23:14.424078 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:14.424061    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa64ef6b-8386-4f0b-b741-b77e4b5d4865-kube-api-access-d728h\" (UniqueName: \"kubernetes.io/projected/fa64ef6b-8386-4f0b-b741-b77e4b5d4865-kube-api-access-d728h\") pod \"fa64ef6b-8386-4f0b-b741-b77e4b5d4865\" (UID: \"fa64ef6b-8386-4f0b-b741-b77e4b5d4865\") "
Oct 30 22:23:14.424240 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:14.424102    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa64ef6b-8386-4f0b-b741-b77e4b5d4865-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa64ef6b-8386-4f0b-b741-b77e4b5d4865-maas-api-xks-ca\") pod \"fa64ef6b-8386-4f0b-b741-b77e4b5d4865\" (UID: \"fa64ef6b-8386-4f0b-b741-b77e4b5d4865\") "
Oct 30 22:23:14.424240 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:14.424171    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa64ef6b-8386-4f0b-b741-b77e4b5d4865-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa64ef6b-8386-4f0b-b741-b77e4b5d4865-maas-api-openshift-ca\") pod \"fa64ef6b-8386-4f0b-b741-b77e4b5d4865\" (UID: \"fa64ef6b-8386-4f0b-b741-b77e4b5d4865\") "
Oct 30 22:23:14.424498 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:14.424412    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73bba646-34ed-4c37-8cdb-df7b4ea77e7f-maas-api-openshift-ca" pod "73bba646-34ed-4c37-8cdb-df7b4ea77e7f" (UID: "73bba646-34ed-4c37-8cdb-df7b4ea77e7f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:23:14.424630 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:14.424591    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa64ef6b-8386-4f0b-b741-b77e4b5d4865-maas-api-openshift-ca" pod "fa64ef6b-8386-4f0b-b741-b77e4b5d4865" (UID: "fa64ef6b-8386-4f0b-b741-b77e4b5d4865"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:23:14.424941 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:14.424920    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa64ef6b-8386-4f0b-b741-b77e4b5d4865-maas-api-openshift-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:14.425040 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:14.424943    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73bba646-34ed-4c37-8cdb-df7b4ea77e7f-maas-api-openshift-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:14.426224 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:14.426175    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa64ef6b-8386-4f0b-b741-b77e4b5d4865-kube-api-access-d728h" pod "fa64ef6b-8386-4f0b-b741-b77e4b5d4865" (UID: "fa64ef6b-8386-4f0b-b741-b77e4b5d4865"). InnerVolumeSpecName "kube-api-access-d728h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:23:14.426363 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:14.426244    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73bba646-34ed-4c37-8cdb-df7b4ea77e7f-maas-api-xks-ca" pod "73bba646-34ed-4c37-8cdb-df7b4ea77e7f" (UID: "73bba646-34ed-4c37-8cdb-df7b4ea77e7f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:23:14.426545 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:14.426507    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa64ef6b-8386-4f0b-b741-b77e4b5d4865-maas-api-xks-ca" pod "fa64ef6b-8386-4f0b-b741-b77e4b5d4865" (UID: "fa64ef6b-8386-4f0b-b741-b77e4b5d4865"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:23:14.426645 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:14.426551    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73bba646-34ed-4c37-8cdb-df7b4ea77e7f-kube-api-access-bx8qn" pod "73bba646-34ed-4c37-8cdb-df7b4ea77e7f" (UID: "73bba646-34ed-4c37-8cdb-df7b4ea77e7f"). InnerVolumeSpecName "kube-api-access-bx8qn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:23:14.525956 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:14.525892    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d728h\" (UniqueName: \"kubernetes.io/projected/fa64ef6b-8386-4f0b-b741-b77e4b5d4865-kube-api-access-d728h\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:14.525956 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:14.525951    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa64ef6b-8386-4f0b-b741-b77e4b5d4865-maas-api-xks-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:14.525956 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:14.525962    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73bba646-34ed-4c37-8cdb-df7b4ea77e7f-maas-api-xks-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:14.526292 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:14.525971    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bx8qn\" (UniqueName: \"kubernetes.io/projected/73bba646-34ed-4c37-8cdb-df7b4ea77e7f-kube-api-access-bx8qn\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:15.177772 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:15.177744    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2d9b5eb-ws4sd"
Oct 30 22:23:15.177979 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:15.177738    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2d9b5eb-ws4sd" event={"ID":"fa64ef6b-8386-4f0b-b741-b77e4b5d4865","Type":"ContainerDied","Data":"f65d5663f1e8534da40ce0a41715b8fa3ae47716356cb6ba95c9de3805b7a361"}
Oct 30 22:23:15.177979 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:15.177867    5061 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f65d5663f1e8534da40ce0a41715b8fa3ae47716356cb6ba95c9de3805b7a361"
Oct 30 22:23:15.179495 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:15.179473    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f29a2d08-qzn4l" event={"ID":"73bba646-34ed-4c37-8cdb-df7b4ea77e7f","Type":"ContainerDied","Data":"83a56d80830e648a8c3e88c65f0505591d86335142091cbe20fec63b7b4992cb"}
Oct 30 22:23:15.179668 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:15.179509    5061 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="83a56d80830e648a8c3e88c65f0505591d86335142091cbe20fec63b7b4992cb"
Oct 30 22:23:15.179668 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:15.179608    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f29a2d08-qzn4l"
Oct 30 22:23:19.844247 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:19.844213    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-5be0d1-6989644d59-xk2lk" podUID="36373ff4-1396-4b98-8f21-8e261b2e0268" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:23:20.217249 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:20.217214    5061 generic.go:356] "Generic (PLEG): container finished" podID="36373ff4-1396-4b98-8f21-8e261b2e0268" containerID="415debea16a93bcde53c7d946768352b6cd0f0878606f3e6a4a7c1f530099f2f" exitCode=0
Oct 30 22:23:20.217372 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:20.217280    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-5be0d1-6989644d59-xk2lk" event={"ID":"36373ff4-1396-4b98-8f21-8e261b2e0268","Type":"ContainerDied","Data":"415debea16a93bcde53c7d946768352b6cd0f0878606f3e6a4a7c1f530099f2f"}
Oct 30 22:23:20.367634 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:20.367580    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-5be0d1-6989644d59-xk2lk"
Oct 30 22:23:20.497053 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:20.497017    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36373ff4-1396-4b98-8f21-8e261b2e0268-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36373ff4-1396-4b98-8f21-8e261b2e0268-plugins-config-volume\") pod \"36373ff4-1396-4b98-8f21-8e261b2e0268\" (UID: \"36373ff4-1396-4b98-8f21-8e261b2e0268\") "
Oct 30 22:23:20.497240 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:20.497069    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36373ff4-1396-4b98-8f21-8e261b2e0268-kube-api-access-7kbrd\" (UniqueName: \"kubernetes.io/projected/36373ff4-1396-4b98-8f21-8e261b2e0268-kube-api-access-7kbrd\") pod \"36373ff4-1396-4b98-8f21-8e261b2e0268\" (UID: \"36373ff4-1396-4b98-8f21-8e261b2e0268\") "
Oct 30 22:23:20.497386 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:20.497360    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36373ff4-1396-4b98-8f21-8e261b2e0268-plugins-config-volume" pod "36373ff4-1396-4b98-8f21-8e261b2e0268" (UID: "36373ff4-1396-4b98-8f21-8e261b2e0268"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:23:20.497661 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:20.497637    5061 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36373ff4-1396-4b98-8f21-8e261b2e0268-plugins-config-volume\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:20.499297 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:20.499266    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36373ff4-1396-4b98-8f21-8e261b2e0268-kube-api-access-7kbrd" pod "36373ff4-1396-4b98-8f21-8e261b2e0268" (UID: "36373ff4-1396-4b98-8f21-8e261b2e0268"). InnerVolumeSpecName "kube-api-access-7kbrd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:23:20.599347 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:20.599318    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kbrd\" (UniqueName: \"kubernetes.io/projected/36373ff4-1396-4b98-8f21-8e261b2e0268-kube-api-access-7kbrd\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:21.228635 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:21.228605    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-5be0d1-6989644d59-xk2lk"
Oct 30 22:23:21.229082 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:21.228652    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-5be0d1-6989644d59-xk2lk" event={"ID":"36373ff4-1396-4b98-8f21-8e261b2e0268","Type":"ContainerDied","Data":"da56f7661420ce624855478867144dd6762771bc69911d3dcafc1e2b4c367241"}
Oct 30 22:23:21.229082 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:21.228706    5061 scope.go:122] "RemoveContainer" containerID="415debea16a93bcde53c7d946768352b6cd0f0878606f3e6a4a7c1f530099f2f"
Oct 30 22:23:21.385332 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:21.385296    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-5be0d1-6989644d59-xk2lk"]
Oct 30 22:23:21.431210 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:21.431179    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-5be0d1-6989644d59-xk2lk"]
Oct 30 22:23:22.529152 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:22.529120    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36373ff4-1396-4b98-8f21-8e261b2e0268" path="/var/lib/kubelet/pods/36373ff4-1396-4b98-8f21-8e261b2e0268/volumes"
Oct 30 22:23:24.806164 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:24.806102    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-0ce563-openshift-default-f75cc7fc9-445r7"]
Oct 30 22:23:24.806719 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:24.806380    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-0ce563-openshift-default-f75cc7fc9-445r7" podUID="0a5f5114-bc2f-48c9-af68-f2d981ec9ce5" containerName="istio-proxy" containerID="cri-o://e44ff32b18531be8417e7c327ccefc08dc7776ab4ab4bc3efff775c162152b97" gracePeriod=30
Oct 30 22:23:24.979577 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:23:24.975263    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 22:23:25.750550 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:25.749004    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2d1f91-kz45z"]
Oct 30 22:23:25.750550 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:25.749904    5061 cpu_manager.go:414] "containerMap: removing container" podUID="73bba646-34ed-4c37-8cdb-df7b4ea77e7f" containerName="revoke-keys"
Oct 30 22:23:25.750550 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:25.749923    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73bba646-34ed-4c37-8cdb-df7b4ea77e7f" containerName="revoke-keys"
Oct 30 22:23:25.750550 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:25.749956    5061 cpu_manager.go:414] "containerMap: removing container" podUID="36373ff4-1396-4b98-8f21-8e261b2e0268" containerName="payload-pre-processing"
Oct 30 22:23:25.750550 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:25.749964    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36373ff4-1396-4b98-8f21-8e261b2e0268" containerName="payload-pre-processing"
Oct 30 22:23:25.750550 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:25.749978    5061 cpu_manager.go:414] "containerMap: removing container" podUID="fa64ef6b-8386-4f0b-b741-b77e4b5d4865" containerName="revoke-keys"
Oct 30 22:23:25.750550 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:25.749987    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa64ef6b-8386-4f0b-b741-b77e4b5d4865" containerName="revoke-keys"
Oct 30 22:23:25.750550 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:25.750156    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="73bba646-34ed-4c37-8cdb-df7b4ea77e7f" containerName="revoke-keys"
Oct 30 22:23:25.750550 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:25.750173    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="36373ff4-1396-4b98-8f21-8e261b2e0268" containerName="payload-pre-processing"
Oct 30 22:23:25.750550 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:25.750186    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa64ef6b-8386-4f0b-b741-b77e4b5d4865" containerName="revoke-keys"
Oct 30 22:23:25.803203 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:25.794686    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2d1f91-kz45z"
Oct 30 22:23:25.856583 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:25.834205    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2d1f91-kz45z"]
Oct 30 22:23:25.888037 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:25.887983    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:23:25.892067 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:25.892042    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gdmfg\""
Oct 30 22:23:25.895851 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:25.895822    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fs7p6\" (UniqueName: \"kubernetes.io/projected/dd2eaa38-7e77-42e1-87c0-dd3b7e0ed0e1-kube-api-access-fs7p6\") pod \"maas-api-revoke-keys-e2e-worker-w0-2d1f91-kz45z\" (UID: \"dd2eaa38-7e77-42e1-87c0-dd3b7e0ed0e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2d1f91-kz45z"
Oct 30 22:23:25.896701 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:25.896676    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd2eaa38-7e77-42e1-87c0-dd3b7e0ed0e1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-2d1f91-kz45z\" (UID: \"dd2eaa38-7e77-42e1-87c0-dd3b7e0ed0e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2d1f91-kz45z"
Oct 30 22:23:25.896885 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:25.896864    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd2eaa38-7e77-42e1-87c0-dd3b7e0ed0e1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-2d1f91-kz45z\" (UID: \"dd2eaa38-7e77-42e1-87c0-dd3b7e0ed0e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2d1f91-kz45z"
Oct 30 22:23:26.001886 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:26.001793    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fs7p6\" (UniqueName: \"kubernetes.io/projected/dd2eaa38-7e77-42e1-87c0-dd3b7e0ed0e1-kube-api-access-fs7p6\") pod \"maas-api-revoke-keys-e2e-worker-w0-2d1f91-kz45z\" (UID: \"dd2eaa38-7e77-42e1-87c0-dd3b7e0ed0e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2d1f91-kz45z"
Oct 30 22:23:26.003414 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:26.003382    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd2eaa38-7e77-42e1-87c0-dd3b7e0ed0e1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-2d1f91-kz45z\" (UID: \"dd2eaa38-7e77-42e1-87c0-dd3b7e0ed0e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2d1f91-kz45z"
Oct 30 22:23:26.025791 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:26.008710    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd2eaa38-7e77-42e1-87c0-dd3b7e0ed0e1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-2d1f91-kz45z\" (UID: \"dd2eaa38-7e77-42e1-87c0-dd3b7e0ed0e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2d1f91-kz45z"
Oct 30 22:23:26.054239 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:26.054211    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd2eaa38-7e77-42e1-87c0-dd3b7e0ed0e1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-2d1f91-kz45z\" (UID: \"dd2eaa38-7e77-42e1-87c0-dd3b7e0ed0e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2d1f91-kz45z"
Oct 30 22:23:26.080507 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:26.080451    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fs7p6\" (UniqueName: \"kubernetes.io/projected/dd2eaa38-7e77-42e1-87c0-dd3b7e0ed0e1-kube-api-access-fs7p6\") pod \"maas-api-revoke-keys-e2e-worker-w0-2d1f91-kz45z\" (UID: \"dd2eaa38-7e77-42e1-87c0-dd3b7e0ed0e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2d1f91-kz45z"
Oct 30 22:23:26.081055 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:26.081031    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd2eaa38-7e77-42e1-87c0-dd3b7e0ed0e1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-2d1f91-kz45z\" (UID: \"dd2eaa38-7e77-42e1-87c0-dd3b7e0ed0e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2d1f91-kz45z"
Oct 30 22:23:26.201240 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:26.201202    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2d1f91-kz45z"
Oct 30 22:23:26.803084 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:26.803044    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2d1f91-kz45z"]
Oct 30 22:23:26.804756 ip-10-0-130-60 kubenswrapper[5061]: W1030 22:23:26.804638    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd2eaa38_7e77_42e1_87c0_dd3b7e0ed0e1.slice/crio-cf75e787248d498e849380cd282a886d14eec145d9a5b481d05a3f5012f0009e WatchSource:0}: Error finding container cf75e787248d498e849380cd282a886d14eec145d9a5b481d05a3f5012f0009e: Status 404 returned error can't find the container with id cf75e787248d498e849380cd282a886d14eec145d9a5b481d05a3f5012f0009e
Oct 30 22:23:27.325632 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:27.325594    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2d1f91-kz45z" event={"ID":"dd2eaa38-7e77-42e1-87c0-dd3b7e0ed0e1","Type":"ContainerStarted","Data":"17dcecf330bfbdde1d1950d44502799ef7f5867ea2d505bf25aaeef3480433c3"}
Oct 30 22:23:27.326042 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:27.325657    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2d1f91-kz45z" event={"ID":"dd2eaa38-7e77-42e1-87c0-dd3b7e0ed0e1","Type":"ContainerStarted","Data":"cf75e787248d498e849380cd282a886d14eec145d9a5b481d05a3f5012f0009e"}
Oct 30 22:23:27.444064 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:27.443959    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2d1f91-kz45z" podStartSLOduration=2.443944365 podStartE2EDuration="2.443944365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:23:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:23:27.44140858 +0000 UTC m=+2461.054178336" watchObservedRunningTime="2026-10-30 22:23:27.443944365 +0000 UTC m=+2461.056714121"
Oct 30 22:23:28.334885 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:28.334843    5061 generic.go:356] "Generic (PLEG): container finished" podID="dd2eaa38-7e77-42e1-87c0-dd3b7e0ed0e1" containerID="17dcecf330bfbdde1d1950d44502799ef7f5867ea2d505bf25aaeef3480433c3" exitCode=0
Oct 30 22:23:28.335305 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:28.334934    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2d1f91-kz45z" event={"ID":"dd2eaa38-7e77-42e1-87c0-dd3b7e0ed0e1","Type":"ContainerDied","Data":"17dcecf330bfbdde1d1950d44502799ef7f5867ea2d505bf25aaeef3480433c3"}
Oct 30 22:23:28.788814 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:28.788732    5061 scope.go:122] "RemoveContainer" containerID="beb09a430bb255c20d2be5aeadc0dd3ef8c5740ab0ad00ae7d56263b90d24f06"
Oct 30 22:23:28.800547 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:28.800512    5061 scope.go:122] "RemoveContainer" containerID="9ead5d84a669e03c1419b5a4f714ee316538a3993a511b0139bfc3f412726020"
Oct 30 22:23:28.812355 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:28.812325    5061 scope.go:122] "RemoveContainer" containerID="c563dc078d52100e9275c06e500dc0100c4e027130e1ed1a1b8474455495e381"
Oct 30 22:23:29.492642 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:29.492616    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2d1f91-kz45z"
Oct 30 22:23:29.592799 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:29.592759    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd2eaa38-7e77-42e1-87c0-dd3b7e0ed0e1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd2eaa38-7e77-42e1-87c0-dd3b7e0ed0e1-maas-api-xks-ca\") pod \"dd2eaa38-7e77-42e1-87c0-dd3b7e0ed0e1\" (UID: \"dd2eaa38-7e77-42e1-87c0-dd3b7e0ed0e1\") "
Oct 30 22:23:29.593006 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:29.592825    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd2eaa38-7e77-42e1-87c0-dd3b7e0ed0e1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd2eaa38-7e77-42e1-87c0-dd3b7e0ed0e1-maas-api-openshift-ca\") pod \"dd2eaa38-7e77-42e1-87c0-dd3b7e0ed0e1\" (UID: \"dd2eaa38-7e77-42e1-87c0-dd3b7e0ed0e1\") "
Oct 30 22:23:29.593006 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:29.592992    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd2eaa38-7e77-42e1-87c0-dd3b7e0ed0e1-kube-api-access-fs7p6\" (UniqueName: \"kubernetes.io/projected/dd2eaa38-7e77-42e1-87c0-dd3b7e0ed0e1-kube-api-access-fs7p6\") pod \"dd2eaa38-7e77-42e1-87c0-dd3b7e0ed0e1\" (UID: \"dd2eaa38-7e77-42e1-87c0-dd3b7e0ed0e1\") "
Oct 30 22:23:29.593162 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:29.593137    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd2eaa38-7e77-42e1-87c0-dd3b7e0ed0e1-maas-api-openshift-ca" pod "dd2eaa38-7e77-42e1-87c0-dd3b7e0ed0e1" (UID: "dd2eaa38-7e77-42e1-87c0-dd3b7e0ed0e1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:23:29.593762 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:29.593733    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd2eaa38-7e77-42e1-87c0-dd3b7e0ed0e1-maas-api-openshift-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:29.595113 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:29.595088    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd2eaa38-7e77-42e1-87c0-dd3b7e0ed0e1-maas-api-xks-ca" pod "dd2eaa38-7e77-42e1-87c0-dd3b7e0ed0e1" (UID: "dd2eaa38-7e77-42e1-87c0-dd3b7e0ed0e1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:23:29.595235 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:29.595220    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd2eaa38-7e77-42e1-87c0-dd3b7e0ed0e1-kube-api-access-fs7p6" pod "dd2eaa38-7e77-42e1-87c0-dd3b7e0ed0e1" (UID: "dd2eaa38-7e77-42e1-87c0-dd3b7e0ed0e1"). InnerVolumeSpecName "kube-api-access-fs7p6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:23:29.694547 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:29.694494    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd2eaa38-7e77-42e1-87c0-dd3b7e0ed0e1-maas-api-xks-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:29.694547 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:29.694522    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fs7p6\" (UniqueName: \"kubernetes.io/projected/dd2eaa38-7e77-42e1-87c0-dd3b7e0ed0e1-kube-api-access-fs7p6\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:30.272706 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:30.228244    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-2d1f91/premium-w0-2d1f91-kserve-6965b558cf-59lgw"]
Oct 30 22:23:30.272706 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:30.236785    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-2d1f91/simulator-w0-2d1f91-kserve-7699765c74-jwtkb"]
Oct 30 22:23:30.456848 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:30.452380    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2d1f91-kz45z" event={"ID":"dd2eaa38-7e77-42e1-87c0-dd3b7e0ed0e1","Type":"ContainerDied","Data":"cf75e787248d498e849380cd282a886d14eec145d9a5b481d05a3f5012f0009e"}
Oct 30 22:23:30.456848 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:30.452445    5061 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cf75e787248d498e849380cd282a886d14eec145d9a5b481d05a3f5012f0009e"
Oct 30 22:23:30.456848 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:30.452682    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2d1f91-kz45z"
Oct 30 22:23:30.508699 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:30.506642    5061 generic.go:356] "Generic (PLEG): container finished" podID="0a5f5114-bc2f-48c9-af68-f2d981ec9ce5" containerID="e44ff32b18531be8417e7c327ccefc08dc7776ab4ab4bc3efff775c162152b97" exitCode=0
Oct 30 22:23:30.508699 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:30.506734    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0ce563-openshift-default-f75cc7fc9-445r7" event={"ID":"0a5f5114-bc2f-48c9-af68-f2d981ec9ce5","Type":"ContainerDied","Data":"e44ff32b18531be8417e7c327ccefc08dc7776ab4ab4bc3efff775c162152b97"}
Oct 30 22:23:30.508699 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:30.506782    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0ce563-openshift-default-f75cc7fc9-445r7" event={"ID":"0a5f5114-bc2f-48c9-af68-f2d981ec9ce5","Type":"ContainerDied","Data":"5e84611aabf2ec82c19d681e131825133046bffd7a6207e0c51849dc3ecb149b"}
Oct 30 22:23:30.508699 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:30.506805    5061 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5e84611aabf2ec82c19d681e131825133046bffd7a6207e0c51849dc3ecb149b"
Oct 30 22:23:30.508699 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:30.506822    5061 scope.go:122] "RemoveContainer" containerID="c26f9976807884d9967ba246acee4e31c5f8093e873cdd5407d6f8f3cde6f64d"
Oct 30 22:23:30.508699 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:30.507653    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-0ce563-openshift-default-f75cc7fc9-445r7"
Oct 30 22:23:30.635223 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:30.635192    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0a5f5114-bc2f-48c9-af68-f2d981ec9ce5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a5f5114-bc2f-48c9-af68-f2d981ec9ce5-istio-podinfo\") pod \"0a5f5114-bc2f-48c9-af68-f2d981ec9ce5\" (UID: \"0a5f5114-bc2f-48c9-af68-f2d981ec9ce5\") "
Oct 30 22:23:30.635223 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:30.635234    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a5f5114-bc2f-48c9-af68-f2d981ec9ce5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a5f5114-bc2f-48c9-af68-f2d981ec9ce5-istio-data\") pod \"0a5f5114-bc2f-48c9-af68-f2d981ec9ce5\" (UID: \"0a5f5114-bc2f-48c9-af68-f2d981ec9ce5\") "
Oct 30 22:23:30.635473 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:30.635348    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a5f5114-bc2f-48c9-af68-f2d981ec9ce5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a5f5114-bc2f-48c9-af68-f2d981ec9ce5-workload-socket\") pod \"0a5f5114-bc2f-48c9-af68-f2d981ec9ce5\" (UID: \"0a5f5114-bc2f-48c9-af68-f2d981ec9ce5\") "
Oct 30 22:23:30.635473 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:30.635444    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a5f5114-bc2f-48c9-af68-f2d981ec9ce5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a5f5114-bc2f-48c9-af68-f2d981ec9ce5-istiod-ca-cert\") pod \"0a5f5114-bc2f-48c9-af68-f2d981ec9ce5\" (UID: \"0a5f5114-bc2f-48c9-af68-f2d981ec9ce5\") "
Oct 30 22:23:30.635602 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:30.635581    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a5f5114-bc2f-48c9-af68-f2d981ec9ce5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a5f5114-bc2f-48c9-af68-f2d981ec9ce5-workload-certs\") pod \"0a5f5114-bc2f-48c9-af68-f2d981ec9ce5\" (UID: \"0a5f5114-bc2f-48c9-af68-f2d981ec9ce5\") "
Oct 30 22:23:30.635702 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:30.635684    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a5f5114-bc2f-48c9-af68-f2d981ec9ce5-kube-api-access-5swlb\" (UniqueName: \"kubernetes.io/projected/0a5f5114-bc2f-48c9-af68-f2d981ec9ce5-kube-api-access-5swlb\") pod \"0a5f5114-bc2f-48c9-af68-f2d981ec9ce5\" (UID: \"0a5f5114-bc2f-48c9-af68-f2d981ec9ce5\") "
Oct 30 22:23:30.635754 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:30.635689    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a5f5114-bc2f-48c9-af68-f2d981ec9ce5-istio-data" pod "0a5f5114-bc2f-48c9-af68-f2d981ec9ce5" (UID: "0a5f5114-bc2f-48c9-af68-f2d981ec9ce5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:23:30.635754 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:30.635731    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a5f5114-bc2f-48c9-af68-f2d981ec9ce5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a5f5114-bc2f-48c9-af68-f2d981ec9ce5-credential-socket\") pod \"0a5f5114-bc2f-48c9-af68-f2d981ec9ce5\" (UID: \"0a5f5114-bc2f-48c9-af68-f2d981ec9ce5\") "
Oct 30 22:23:30.635848 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:30.635799    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a5f5114-bc2f-48c9-af68-f2d981ec9ce5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a5f5114-bc2f-48c9-af68-f2d981ec9ce5-istio-envoy\") pod \"0a5f5114-bc2f-48c9-af68-f2d981ec9ce5\" (UID: \"0a5f5114-bc2f-48c9-af68-f2d981ec9ce5\") "
Oct 30 22:23:30.635895 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:30.635849    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a5f5114-bc2f-48c9-af68-f2d981ec9ce5-istio-token\" (UniqueName: \"kubernetes.io/projected/0a5f5114-bc2f-48c9-af68-f2d981ec9ce5-istio-token\") pod \"0a5f5114-bc2f-48c9-af68-f2d981ec9ce5\" (UID: \"0a5f5114-bc2f-48c9-af68-f2d981ec9ce5\") "
Oct 30 22:23:30.635895 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:30.635863    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a5f5114-bc2f-48c9-af68-f2d981ec9ce5-workload-certs" pod "0a5f5114-bc2f-48c9-af68-f2d981ec9ce5" (UID: "0a5f5114-bc2f-48c9-af68-f2d981ec9ce5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:23:30.636029 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:30.635929    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a5f5114-bc2f-48c9-af68-f2d981ec9ce5-istiod-ca-cert" pod "0a5f5114-bc2f-48c9-af68-f2d981ec9ce5" (UID: "0a5f5114-bc2f-48c9-af68-f2d981ec9ce5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:23:30.636191 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:30.636172    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a5f5114-bc2f-48c9-af68-f2d981ec9ce5-credential-socket" pod "0a5f5114-bc2f-48c9-af68-f2d981ec9ce5" (UID: "0a5f5114-bc2f-48c9-af68-f2d981ec9ce5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:23:30.636376 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:30.636356    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a5f5114-bc2f-48c9-af68-f2d981ec9ce5-workload-socket" pod "0a5f5114-bc2f-48c9-af68-f2d981ec9ce5" (UID: "0a5f5114-bc2f-48c9-af68-f2d981ec9ce5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:23:30.636690 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:30.636668    5061 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a5f5114-bc2f-48c9-af68-f2d981ec9ce5-workload-socket\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:30.636797 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:30.636694    5061 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a5f5114-bc2f-48c9-af68-f2d981ec9ce5-istiod-ca-cert\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:30.636797 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:30.636708    5061 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a5f5114-bc2f-48c9-af68-f2d981ec9ce5-workload-certs\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:30.636797 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:30.636721    5061 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a5f5114-bc2f-48c9-af68-f2d981ec9ce5-credential-socket\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:30.636797 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:30.636733    5061 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a5f5114-bc2f-48c9-af68-f2d981ec9ce5-istio-data\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:30.638307 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:30.638274    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a5f5114-bc2f-48c9-af68-f2d981ec9ce5-istio-envoy" pod "0a5f5114-bc2f-48c9-af68-f2d981ec9ce5" (UID: "0a5f5114-bc2f-48c9-af68-f2d981ec9ce5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:23:30.638969 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:30.638946    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a5f5114-bc2f-48c9-af68-f2d981ec9ce5-istio-token" pod "0a5f5114-bc2f-48c9-af68-f2d981ec9ce5" (UID: "0a5f5114-bc2f-48c9-af68-f2d981ec9ce5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:23:30.641562 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:30.639800    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a5f5114-bc2f-48c9-af68-f2d981ec9ce5-kube-api-access-5swlb" pod "0a5f5114-bc2f-48c9-af68-f2d981ec9ce5" (UID: "0a5f5114-bc2f-48c9-af68-f2d981ec9ce5"). InnerVolumeSpecName "kube-api-access-5swlb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:23:30.641562 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:30.639802    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0a5f5114-bc2f-48c9-af68-f2d981ec9ce5-istio-podinfo" pod "0a5f5114-bc2f-48c9-af68-f2d981ec9ce5" (UID: "0a5f5114-bc2f-48c9-af68-f2d981ec9ce5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:23:30.738561 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:30.738507    5061 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a5f5114-bc2f-48c9-af68-f2d981ec9ce5-istio-token\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:30.738561 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:30.738562    5061 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a5f5114-bc2f-48c9-af68-f2d981ec9ce5-istio-podinfo\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:30.738819 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:30.738580    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5swlb\" (UniqueName: \"kubernetes.io/projected/0a5f5114-bc2f-48c9-af68-f2d981ec9ce5-kube-api-access-5swlb\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:30.738819 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:30.738597    5061 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a5f5114-bc2f-48c9-af68-f2d981ec9ce5-istio-envoy\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:31.389330 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:31.389293    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-c1e2ab-openshift-default-bcd8478cf-jg8sp"]
Oct 30 22:23:31.389997 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:31.389964    5061 cpu_manager.go:414] "containerMap: removing container" podUID="0a5f5114-bc2f-48c9-af68-f2d981ec9ce5" containerName="istio-proxy"
Oct 30 22:23:31.389997 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:31.389985    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a5f5114-bc2f-48c9-af68-f2d981ec9ce5" containerName="istio-proxy"
Oct 30 22:23:31.390202 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:31.390008    5061 cpu_manager.go:414] "containerMap: removing container" podUID="0a5f5114-bc2f-48c9-af68-f2d981ec9ce5" containerName="istio-proxy"
Oct 30 22:23:31.390202 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:31.390035    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a5f5114-bc2f-48c9-af68-f2d981ec9ce5" containerName="istio-proxy"
Oct 30 22:23:31.390202 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:31.390106    5061 cpu_manager.go:414] "containerMap: removing container" podUID="dd2eaa38-7e77-42e1-87c0-dd3b7e0ed0e1" containerName="revoke-keys"
Oct 30 22:23:31.390202 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:31.390114    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd2eaa38-7e77-42e1-87c0-dd3b7e0ed0e1" containerName="revoke-keys"
Oct 30 22:23:31.390425 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:31.390261    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a5f5114-bc2f-48c9-af68-f2d981ec9ce5" containerName="istio-proxy"
Oct 30 22:23:31.390425 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:31.390276    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a5f5114-bc2f-48c9-af68-f2d981ec9ce5" containerName="istio-proxy"
Oct 30 22:23:31.390425 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:31.390302    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd2eaa38-7e77-42e1-87c0-dd3b7e0ed0e1" containerName="revoke-keys"
Oct 30 22:23:31.395103 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:31.395082    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c1e2ab-openshift-default-bcd8478cf-jg8sp"
Oct 30 22:23:31.445899 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:31.445876    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-c1e2ab-openshift-default-dockercfg-rh8s2\""
Oct 30 22:23:31.447439 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:31.447414    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea11c8ac-4a65-463d-8956-739f3ae20611-istio-podinfo\") pod \"e2e-api-b-c1e2ab-openshift-default-bcd8478cf-jg8sp\" (UID: \"ea11c8ac-4a65-463d-8956-739f3ae20611\") " pod="openshift-ingress/e2e-api-b-c1e2ab-openshift-default-bcd8478cf-jg8sp"
Oct 30 22:23:31.447626 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:31.447610    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea11c8ac-4a65-463d-8956-739f3ae20611-istiod-ca-cert\") pod \"e2e-api-b-c1e2ab-openshift-default-bcd8478cf-jg8sp\" (UID: \"ea11c8ac-4a65-463d-8956-739f3ae20611\") " pod="openshift-ingress/e2e-api-b-c1e2ab-openshift-default-bcd8478cf-jg8sp"
Oct 30 22:23:31.447669 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:31.447642    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea11c8ac-4a65-463d-8956-739f3ae20611-workload-certs\") pod \"e2e-api-b-c1e2ab-openshift-default-bcd8478cf-jg8sp\" (UID: \"ea11c8ac-4a65-463d-8956-739f3ae20611\") " pod="openshift-ingress/e2e-api-b-c1e2ab-openshift-default-bcd8478cf-jg8sp"
Oct 30 22:23:31.447669 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:31.447660    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea11c8ac-4a65-463d-8956-739f3ae20611-credential-socket\") pod \"e2e-api-b-c1e2ab-openshift-default-bcd8478cf-jg8sp\" (UID: \"ea11c8ac-4a65-463d-8956-739f3ae20611\") " pod="openshift-ingress/e2e-api-b-c1e2ab-openshift-default-bcd8478cf-jg8sp"
Oct 30 22:23:31.447730 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:31.447707    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4nwz\" (UniqueName: \"kubernetes.io/projected/ea11c8ac-4a65-463d-8956-739f3ae20611-kube-api-access-s4nwz\") pod \"e2e-api-b-c1e2ab-openshift-default-bcd8478cf-jg8sp\" (UID: \"ea11c8ac-4a65-463d-8956-739f3ae20611\") " pod="openshift-ingress/e2e-api-b-c1e2ab-openshift-default-bcd8478cf-jg8sp"
Oct 30 22:23:31.447803 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:31.447791    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea11c8ac-4a65-463d-8956-739f3ae20611-istio-envoy\") pod \"e2e-api-b-c1e2ab-openshift-default-bcd8478cf-jg8sp\" (UID: \"ea11c8ac-4a65-463d-8956-739f3ae20611\") " pod="openshift-ingress/e2e-api-b-c1e2ab-openshift-default-bcd8478cf-jg8sp"
Oct 30 22:23:31.447845 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:31.447835    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea11c8ac-4a65-463d-8956-739f3ae20611-istio-data\") pod \"e2e-api-b-c1e2ab-openshift-default-bcd8478cf-jg8sp\" (UID: \"ea11c8ac-4a65-463d-8956-739f3ae20611\") " pod="openshift-ingress/e2e-api-b-c1e2ab-openshift-default-bcd8478cf-jg8sp"
Oct 30 22:23:31.447895 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:31.447854    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea11c8ac-4a65-463d-8956-739f3ae20611-workload-socket\") pod \"e2e-api-b-c1e2ab-openshift-default-bcd8478cf-jg8sp\" (UID: \"ea11c8ac-4a65-463d-8956-739f3ae20611\") " pod="openshift-ingress/e2e-api-b-c1e2ab-openshift-default-bcd8478cf-jg8sp"
Oct 30 22:23:31.447958 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:31.447945    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea11c8ac-4a65-463d-8956-739f3ae20611-istio-token\") pod \"e2e-api-b-c1e2ab-openshift-default-bcd8478cf-jg8sp\" (UID: \"ea11c8ac-4a65-463d-8956-739f3ae20611\") " pod="openshift-ingress/e2e-api-b-c1e2ab-openshift-default-bcd8478cf-jg8sp"
Oct 30 22:23:31.483076 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:31.483044    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-c1e2ab-openshift-default-bcd8478cf-jg8sp"]
Oct 30 22:23:31.530709 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:31.530680    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-0ce563-openshift-default-f75cc7fc9-445r7"
Oct 30 22:23:31.548745 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:31.548719    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea11c8ac-4a65-463d-8956-739f3ae20611-istio-podinfo\") pod \"e2e-api-b-c1e2ab-openshift-default-bcd8478cf-jg8sp\" (UID: \"ea11c8ac-4a65-463d-8956-739f3ae20611\") " pod="openshift-ingress/e2e-api-b-c1e2ab-openshift-default-bcd8478cf-jg8sp"
Oct 30 22:23:31.548845 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:31.548800    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea11c8ac-4a65-463d-8956-739f3ae20611-istiod-ca-cert\") pod \"e2e-api-b-c1e2ab-openshift-default-bcd8478cf-jg8sp\" (UID: \"ea11c8ac-4a65-463d-8956-739f3ae20611\") " pod="openshift-ingress/e2e-api-b-c1e2ab-openshift-default-bcd8478cf-jg8sp"
Oct 30 22:23:31.548845 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:31.548820    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea11c8ac-4a65-463d-8956-739f3ae20611-workload-certs\") pod \"e2e-api-b-c1e2ab-openshift-default-bcd8478cf-jg8sp\" (UID: \"ea11c8ac-4a65-463d-8956-739f3ae20611\") " pod="openshift-ingress/e2e-api-b-c1e2ab-openshift-default-bcd8478cf-jg8sp"
Oct 30 22:23:31.548845 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:31.548834    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea11c8ac-4a65-463d-8956-739f3ae20611-credential-socket\") pod \"e2e-api-b-c1e2ab-openshift-default-bcd8478cf-jg8sp\" (UID: \"ea11c8ac-4a65-463d-8956-739f3ae20611\") " pod="openshift-ingress/e2e-api-b-c1e2ab-openshift-default-bcd8478cf-jg8sp"
Oct 30 22:23:31.548976 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:31.548851    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4nwz\" (UniqueName: \"kubernetes.io/projected/ea11c8ac-4a65-463d-8956-739f3ae20611-kube-api-access-s4nwz\") pod \"e2e-api-b-c1e2ab-openshift-default-bcd8478cf-jg8sp\" (UID: \"ea11c8ac-4a65-463d-8956-739f3ae20611\") " pod="openshift-ingress/e2e-api-b-c1e2ab-openshift-default-bcd8478cf-jg8sp"
Oct 30 22:23:31.548976 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:31.548898    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea11c8ac-4a65-463d-8956-739f3ae20611-istio-envoy\") pod \"e2e-api-b-c1e2ab-openshift-default-bcd8478cf-jg8sp\" (UID: \"ea11c8ac-4a65-463d-8956-739f3ae20611\") " pod="openshift-ingress/e2e-api-b-c1e2ab-openshift-default-bcd8478cf-jg8sp"
Oct 30 22:23:31.548976 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:31.548947    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea11c8ac-4a65-463d-8956-739f3ae20611-istio-data\") pod \"e2e-api-b-c1e2ab-openshift-default-bcd8478cf-jg8sp\" (UID: \"ea11c8ac-4a65-463d-8956-739f3ae20611\") " pod="openshift-ingress/e2e-api-b-c1e2ab-openshift-default-bcd8478cf-jg8sp"
Oct 30 22:23:31.548976 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:31.548973    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea11c8ac-4a65-463d-8956-739f3ae20611-workload-socket\") pod \"e2e-api-b-c1e2ab-openshift-default-bcd8478cf-jg8sp\" (UID: \"ea11c8ac-4a65-463d-8956-739f3ae20611\") " pod="openshift-ingress/e2e-api-b-c1e2ab-openshift-default-bcd8478cf-jg8sp"
Oct 30 22:23:31.549173 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:31.549026    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea11c8ac-4a65-463d-8956-739f3ae20611-istio-token\") pod \"e2e-api-b-c1e2ab-openshift-default-bcd8478cf-jg8sp\" (UID: \"ea11c8ac-4a65-463d-8956-739f3ae20611\") " pod="openshift-ingress/e2e-api-b-c1e2ab-openshift-default-bcd8478cf-jg8sp"
Oct 30 22:23:31.549410 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:31.549386    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea11c8ac-4a65-463d-8956-739f3ae20611-workload-certs\") pod \"e2e-api-b-c1e2ab-openshift-default-bcd8478cf-jg8sp\" (UID: \"ea11c8ac-4a65-463d-8956-739f3ae20611\") " pod="openshift-ingress/e2e-api-b-c1e2ab-openshift-default-bcd8478cf-jg8sp"
Oct 30 22:23:31.549496 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:31.549454    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea11c8ac-4a65-463d-8956-739f3ae20611-credential-socket\") pod \"e2e-api-b-c1e2ab-openshift-default-bcd8478cf-jg8sp\" (UID: \"ea11c8ac-4a65-463d-8956-739f3ae20611\") " pod="openshift-ingress/e2e-api-b-c1e2ab-openshift-default-bcd8478cf-jg8sp"
Oct 30 22:23:31.549586 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:31.549497    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea11c8ac-4a65-463d-8956-739f3ae20611-workload-socket\") pod \"e2e-api-b-c1e2ab-openshift-default-bcd8478cf-jg8sp\" (UID: \"ea11c8ac-4a65-463d-8956-739f3ae20611\") " pod="openshift-ingress/e2e-api-b-c1e2ab-openshift-default-bcd8478cf-jg8sp"
Oct 30 22:23:31.549645 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:31.549588    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea11c8ac-4a65-463d-8956-739f3ae20611-istio-data\") pod \"e2e-api-b-c1e2ab-openshift-default-bcd8478cf-jg8sp\" (UID: \"ea11c8ac-4a65-463d-8956-739f3ae20611\") " pod="openshift-ingress/e2e-api-b-c1e2ab-openshift-default-bcd8478cf-jg8sp"
Oct 30 22:23:31.549761 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:31.549736    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea11c8ac-4a65-463d-8956-739f3ae20611-istiod-ca-cert\") pod \"e2e-api-b-c1e2ab-openshift-default-bcd8478cf-jg8sp\" (UID: \"ea11c8ac-4a65-463d-8956-739f3ae20611\") " pod="openshift-ingress/e2e-api-b-c1e2ab-openshift-default-bcd8478cf-jg8sp"
Oct 30 22:23:31.551145 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:31.551124    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea11c8ac-4a65-463d-8956-739f3ae20611-istio-envoy\") pod \"e2e-api-b-c1e2ab-openshift-default-bcd8478cf-jg8sp\" (UID: \"ea11c8ac-4a65-463d-8956-739f3ae20611\") " pod="openshift-ingress/e2e-api-b-c1e2ab-openshift-default-bcd8478cf-jg8sp"
Oct 30 22:23:31.551397 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:31.551379    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea11c8ac-4a65-463d-8956-739f3ae20611-istio-podinfo\") pod \"e2e-api-b-c1e2ab-openshift-default-bcd8478cf-jg8sp\" (UID: \"ea11c8ac-4a65-463d-8956-739f3ae20611\") " pod="openshift-ingress/e2e-api-b-c1e2ab-openshift-default-bcd8478cf-jg8sp"
Oct 30 22:23:31.605865 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:31.605836    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea11c8ac-4a65-463d-8956-739f3ae20611-istio-token\") pod \"e2e-api-b-c1e2ab-openshift-default-bcd8478cf-jg8sp\" (UID: \"ea11c8ac-4a65-463d-8956-739f3ae20611\") " pod="openshift-ingress/e2e-api-b-c1e2ab-openshift-default-bcd8478cf-jg8sp"
Oct 30 22:23:31.606013 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:31.605985    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4nwz\" (UniqueName: \"kubernetes.io/projected/ea11c8ac-4a65-463d-8956-739f3ae20611-kube-api-access-s4nwz\") pod \"e2e-api-b-c1e2ab-openshift-default-bcd8478cf-jg8sp\" (UID: \"ea11c8ac-4a65-463d-8956-739f3ae20611\") " pod="openshift-ingress/e2e-api-b-c1e2ab-openshift-default-bcd8478cf-jg8sp"
Oct 30 22:23:31.708825 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:31.708751    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c1e2ab-openshift-default-bcd8478cf-jg8sp"
Oct 30 22:23:31.753143 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:31.753108    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-0ce563-openshift-default-f75cc7fc9-445r7"]
Oct 30 22:23:31.756679 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:31.756651    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-0ce563-openshift-default-f75cc7fc9-445r7"]
Oct 30 22:23:32.126840 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:32.126807    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-c1e2ab-openshift-default-bcd8478cf-jg8sp"]
Oct 30 22:23:32.128038 ip-10-0-130-60 kubenswrapper[5061]: W1030 22:23:32.128010    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea11c8ac_4a65_463d_8956_739f3ae20611.slice/crio-cc593246e8c0f6f2d66b58031fa1fe4561c2ba224a8a7eb0b4e523f864dc3c43 WatchSource:0}: Error finding container cc593246e8c0f6f2d66b58031fa1fe4561c2ba224a8a7eb0b4e523f864dc3c43: Status 404 returned error can't find the container with id cc593246e8c0f6f2d66b58031fa1fe4561c2ba224a8a7eb0b4e523f864dc3c43
Oct 30 22:23:32.131736 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:32.131705    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:23:32.131839 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:32.131769    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:23:32.131839 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:32.131797    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:23:32.528911 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:32.528821    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a5f5114-bc2f-48c9-af68-f2d981ec9ce5" path="/var/lib/kubelet/pods/0a5f5114-bc2f-48c9-af68-f2d981ec9ce5/volumes"
Oct 30 22:23:32.542594 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:32.542562    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c1e2ab-openshift-default-bcd8478cf-jg8sp" event={"ID":"ea11c8ac-4a65-463d-8956-739f3ae20611","Type":"ContainerStarted","Data":"63e08c843faa7a7f6333e45958d7c45f971dc9d933d75456abf77004208375ad"}
Oct 30 22:23:32.543001 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:32.542620    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c1e2ab-openshift-default-bcd8478cf-jg8sp" event={"ID":"ea11c8ac-4a65-463d-8956-739f3ae20611","Type":"ContainerStarted","Data":"cc593246e8c0f6f2d66b58031fa1fe4561c2ba224a8a7eb0b4e523f864dc3c43"}
Oct 30 22:23:32.730551 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:32.730005    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-c1e2ab-openshift-default-bcd8478cf-jg8sp"
Oct 30 22:23:32.754211 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:32.754150    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-c1e2ab-openshift-default-bcd8478cf-jg8sp" podStartSLOduration=1.75412922 podStartE2EDuration="1.75412922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:23:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:23:32.702861157 +0000 UTC m=+2466.315630916" watchObservedRunningTime="2026-10-30 22:23:32.75412922 +0000 UTC m=+2466.366898979"
Oct 30 22:23:33.732768 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:33.732729    5061 patch_prober.go:29] interesting pod/e2e-api-b-c1e2ab-openshift-default-bcd8478cf-jg8sp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body=
Oct 30 22:23:33.733187 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:33.732792    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c1e2ab-openshift-default-bcd8478cf-jg8sp" podUID="ea11c8ac-4a65-463d-8956-739f3ae20611" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused"
Oct 30 22:23:34.717898 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:34.717863    5061 patch_prober.go:29] interesting pod/e2e-api-b-c1e2ab-openshift-default-bcd8478cf-jg8sp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body=
Oct 30 22:23:34.718172 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:34.718143    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c1e2ab-openshift-default-bcd8478cf-jg8sp" podUID="ea11c8ac-4a65-463d-8956-739f3ae20611" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused"
Oct 30 22:23:35.729552 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:35.728948    5061 patch_prober.go:29] interesting pod/e2e-api-b-c1e2ab-openshift-default-bcd8478cf-jg8sp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body=
Oct 30 22:23:35.729552 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:35.729021    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c1e2ab-openshift-default-bcd8478cf-jg8sp" podUID="ea11c8ac-4a65-463d-8956-739f3ae20611" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused"
Oct 30 22:23:36.709925 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:36.709884    5061 patch_prober.go:29] interesting pod/e2e-api-b-c1e2ab-openshift-default-bcd8478cf-jg8sp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body=
Oct 30 22:23:36.710215 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:36.710183    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c1e2ab-openshift-default-bcd8478cf-jg8sp" podUID="ea11c8ac-4a65-463d-8956-739f3ae20611" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused"
Oct 30 22:23:37.721883 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:37.721848    5061 patch_prober.go:29] interesting pod/e2e-api-b-c1e2ab-openshift-default-bcd8478cf-jg8sp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body=
Oct 30 22:23:37.722462 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:37.722429    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c1e2ab-openshift-default-bcd8478cf-jg8sp" podUID="ea11c8ac-4a65-463d-8956-739f3ae20611" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused"
Oct 30 22:23:37.886253 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:37.886214    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-2d1f91-df6cc69d8-wzwll"]
Oct 30 22:23:38.714187 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:38.714115    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-c1e2ab-openshift-default-bcd8478cf-jg8sp"
Oct 30 22:23:38.714416 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:38.714392    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-c1e2ab-openshift-default-bcd8478cf-jg8sp"
Oct 30 22:23:38.715203 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:38.715182    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-c1e2ab-openshift-default-bcd8478cf-jg8sp"
Oct 30 22:23:42.887081 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:42.887034    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-2d1f91-df6cc69d8-wzwll" podUID="d71ce3d8-00ba-4ff3-a870-6dd143dbaf93" containerName="payload-processing" containerID="cri-o://41f367b9e84d1295d677672d630e913cd2bff627bee86bd938a7d31706bc6628" gracePeriod=25
Oct 30 22:23:43.149240 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:43.149185    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-2d1f91-df6cc69d8-wzwll"
Oct 30 22:23:43.223148 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:43.223113    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d71ce3d8-00ba-4ff3-a870-6dd143dbaf93-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d71ce3d8-00ba-4ff3-a870-6dd143dbaf93-plugins-config-volume\") pod \"d71ce3d8-00ba-4ff3-a870-6dd143dbaf93\" (UID: \"d71ce3d8-00ba-4ff3-a870-6dd143dbaf93\") "
Oct 30 22:23:43.223298 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:43.223239    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d71ce3d8-00ba-4ff3-a870-6dd143dbaf93-kube-api-access-zxpf4\" (UniqueName: \"kubernetes.io/projected/d71ce3d8-00ba-4ff3-a870-6dd143dbaf93-kube-api-access-zxpf4\") pod \"d71ce3d8-00ba-4ff3-a870-6dd143dbaf93\" (UID: \"d71ce3d8-00ba-4ff3-a870-6dd143dbaf93\") "
Oct 30 22:23:43.223467 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:43.223444    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d71ce3d8-00ba-4ff3-a870-6dd143dbaf93-plugins-config-volume" pod "d71ce3d8-00ba-4ff3-a870-6dd143dbaf93" (UID: "d71ce3d8-00ba-4ff3-a870-6dd143dbaf93"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:23:43.223676 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:43.223661    5061 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d71ce3d8-00ba-4ff3-a870-6dd143dbaf93-plugins-config-volume\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:43.225314 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:43.225293    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d71ce3d8-00ba-4ff3-a870-6dd143dbaf93-kube-api-access-zxpf4" pod "d71ce3d8-00ba-4ff3-a870-6dd143dbaf93" (UID: "d71ce3d8-00ba-4ff3-a870-6dd143dbaf93"). InnerVolumeSpecName "kube-api-access-zxpf4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:23:43.324505 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:43.324472    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxpf4\" (UniqueName: \"kubernetes.io/projected/d71ce3d8-00ba-4ff3-a870-6dd143dbaf93-kube-api-access-zxpf4\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:43.619521 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:43.619488    5061 generic.go:356] "Generic (PLEG): container finished" podID="d71ce3d8-00ba-4ff3-a870-6dd143dbaf93" containerID="41f367b9e84d1295d677672d630e913cd2bff627bee86bd938a7d31706bc6628" exitCode=0
Oct 30 22:23:43.619700 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:43.619571    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-2d1f91-df6cc69d8-wzwll"
Oct 30 22:23:43.619700 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:43.619568    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-2d1f91-df6cc69d8-wzwll" event={"ID":"d71ce3d8-00ba-4ff3-a870-6dd143dbaf93","Type":"ContainerDied","Data":"41f367b9e84d1295d677672d630e913cd2bff627bee86bd938a7d31706bc6628"}
Oct 30 22:23:43.619700 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:43.619683    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-2d1f91-df6cc69d8-wzwll" event={"ID":"d71ce3d8-00ba-4ff3-a870-6dd143dbaf93","Type":"ContainerDied","Data":"fc1dc6f7f21aa0ae9b1c8c2fd38d539ee8c72447d6c722dea7e193aeddf4367d"}
Oct 30 22:23:43.619837 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:43.619705    5061 scope.go:122] "RemoveContainer" containerID="41f367b9e84d1295d677672d630e913cd2bff627bee86bd938a7d31706bc6628"
Oct 30 22:23:43.630575 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:43.630518    5061 scope.go:122] "RemoveContainer" containerID="41f367b9e84d1295d677672d630e913cd2bff627bee86bd938a7d31706bc6628"
Oct 30 22:23:43.630837 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:23:43.630820    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41f367b9e84d1295d677672d630e913cd2bff627bee86bd938a7d31706bc6628\": container with ID starting with 41f367b9e84d1295d677672d630e913cd2bff627bee86bd938a7d31706bc6628 not found: ID does not exist" containerID="41f367b9e84d1295d677672d630e913cd2bff627bee86bd938a7d31706bc6628"
Oct 30 22:23:43.630926 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:43.630844    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41f367b9e84d1295d677672d630e913cd2bff627bee86bd938a7d31706bc6628"} err="failed to get container status \"41f367b9e84d1295d677672d630e913cd2bff627bee86bd938a7d31706bc6628\": rpc error: code = NotFound desc = could not find container \"41f367b9e84d1295d677672d630e913cd2bff627bee86bd938a7d31706bc6628\": container with ID starting with 41f367b9e84d1295d677672d630e913cd2bff627bee86bd938a7d31706bc6628 not found: ID does not exist"
Oct 30 22:23:43.779986 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:43.779947    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-2d1f91-df6cc69d8-wzwll"]
Oct 30 22:23:43.783243 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:43.783214    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-2d1f91-df6cc69d8-wzwll"]
Oct 30 22:23:44.528867 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:44.528834    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d71ce3d8-00ba-4ff3-a870-6dd143dbaf93" path="/var/lib/kubelet/pods/d71ce3d8-00ba-4ff3-a870-6dd143dbaf93/volumes"
Oct 30 22:23:45.365145 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:45.365106    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-2d1f91/simulator-w0-2d1f91-kserve-7699765c74-jwtkb" podUID="44b7d6f6-32d8-4ef4-95bc-eef604fa6b5b" containerName="main" containerID="cri-o://5d169ae8c76d09ccb1d3892d416f889929adf8f367b589272f0a20ec1d5e5c97" gracePeriod=45
Oct 30 22:23:45.427623 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:45.427591    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-2d1f91/premium-w0-2d1f91-kserve-6965b558cf-59lgw" podUID="fb84233d-25e0-48bc-8899-c3a3cb3cfeb9" containerName="main" containerID="cri-o://161ec0d5ec48158bd3d0eac0af1fc29538a01f4d1682c350cb8e36c822315f22" gracePeriod=45
Oct 30 22:23:45.617099 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:45.616702    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2d1f91/premium-w0-2d1f91-kserve-6965b558cf-59lgw" podUID="fb84233d-25e0-48bc-8899-c3a3cb3cfeb9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": dial tcp 10.132.0.69:8000: connect: connection refused"
Oct 30 22:23:45.642536 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:45.642497    5061 generic.go:356] "Generic (PLEG): container finished" podID="fb84233d-25e0-48bc-8899-c3a3cb3cfeb9" containerID="161ec0d5ec48158bd3d0eac0af1fc29538a01f4d1682c350cb8e36c822315f22" exitCode=0
Oct 30 22:23:45.642692 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:45.642586    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2d1f91/premium-w0-2d1f91-kserve-6965b558cf-59lgw" event={"ID":"fb84233d-25e0-48bc-8899-c3a3cb3cfeb9","Type":"ContainerDied","Data":"161ec0d5ec48158bd3d0eac0af1fc29538a01f4d1682c350cb8e36c822315f22"}
Oct 30 22:23:45.644640 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:45.644603    5061 generic.go:356] "Generic (PLEG): container finished" podID="44b7d6f6-32d8-4ef4-95bc-eef604fa6b5b" containerID="5d169ae8c76d09ccb1d3892d416f889929adf8f367b589272f0a20ec1d5e5c97" exitCode=0
Oct 30 22:23:45.644799 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:45.644684    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2d1f91/simulator-w0-2d1f91-kserve-7699765c74-jwtkb" event={"ID":"44b7d6f6-32d8-4ef4-95bc-eef604fa6b5b","Type":"ContainerDied","Data":"5d169ae8c76d09ccb1d3892d416f889929adf8f367b589272f0a20ec1d5e5c97"}
Oct 30 22:23:45.731789 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:45.731762    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2d1f91/simulator-w0-2d1f91-kserve-7699765c74-jwtkb"
Oct 30 22:23:45.735577 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:45.735556    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2d1f91/premium-w0-2d1f91-kserve-6965b558cf-59lgw"
Oct 30 22:23:45.752591 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:45.752513    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44b7d6f6-32d8-4ef4-95bc-eef604fa6b5b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44b7d6f6-32d8-4ef4-95bc-eef604fa6b5b-model-cache\") pod \"44b7d6f6-32d8-4ef4-95bc-eef604fa6b5b\" (UID: \"44b7d6f6-32d8-4ef4-95bc-eef604fa6b5b\") "
Oct 30 22:23:45.752765 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:45.752595    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb84233d-25e0-48bc-8899-c3a3cb3cfeb9-kube-api-access-8z9h2\" (UniqueName: \"kubernetes.io/projected/fb84233d-25e0-48bc-8899-c3a3cb3cfeb9-kube-api-access-8z9h2\") pod \"fb84233d-25e0-48bc-8899-c3a3cb3cfeb9\" (UID: \"fb84233d-25e0-48bc-8899-c3a3cb3cfeb9\") "
Oct 30 22:23:45.752765 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:45.752657    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44b7d6f6-32d8-4ef4-95bc-eef604fa6b5b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44b7d6f6-32d8-4ef4-95bc-eef604fa6b5b-tmp-dir\") pod \"44b7d6f6-32d8-4ef4-95bc-eef604fa6b5b\" (UID: \"44b7d6f6-32d8-4ef4-95bc-eef604fa6b5b\") "
Oct 30 22:23:45.752765 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:45.752689    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44b7d6f6-32d8-4ef4-95bc-eef604fa6b5b-kube-api-access-pg9wx\" (UniqueName: \"kubernetes.io/projected/44b7d6f6-32d8-4ef4-95bc-eef604fa6b5b-kube-api-access-pg9wx\") pod \"44b7d6f6-32d8-4ef4-95bc-eef604fa6b5b\" (UID: \"44b7d6f6-32d8-4ef4-95bc-eef604fa6b5b\") "
Oct 30 22:23:45.752765 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:45.752716    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb84233d-25e0-48bc-8899-c3a3cb3cfeb9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb84233d-25e0-48bc-8899-c3a3cb3cfeb9-model-cache\") pod \"fb84233d-25e0-48bc-8899-c3a3cb3cfeb9\" (UID: \"fb84233d-25e0-48bc-8899-c3a3cb3cfeb9\") "
Oct 30 22:23:45.752983 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:45.752766    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44b7d6f6-32d8-4ef4-95bc-eef604fa6b5b-model-cache" pod "44b7d6f6-32d8-4ef4-95bc-eef604fa6b5b" (UID: "44b7d6f6-32d8-4ef4-95bc-eef604fa6b5b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:23:45.752983 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:45.752781    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44b7d6f6-32d8-4ef4-95bc-eef604fa6b5b-home\" (UniqueName: \"kubernetes.io/empty-dir/44b7d6f6-32d8-4ef4-95bc-eef604fa6b5b-home\") pod \"44b7d6f6-32d8-4ef4-95bc-eef604fa6b5b\" (UID: \"44b7d6f6-32d8-4ef4-95bc-eef604fa6b5b\") "
Oct 30 22:23:45.752983 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:45.752810    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/44b7d6f6-32d8-4ef4-95bc-eef604fa6b5b-tls-certs\" (UniqueName: \"kubernetes.io/secret/44b7d6f6-32d8-4ef4-95bc-eef604fa6b5b-tls-certs\") pod \"44b7d6f6-32d8-4ef4-95bc-eef604fa6b5b\" (UID: \"44b7d6f6-32d8-4ef4-95bc-eef604fa6b5b\") "
Oct 30 22:23:45.752983 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:45.752886    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb84233d-25e0-48bc-8899-c3a3cb3cfeb9-tls-certs\" (UniqueName: \"kubernetes.io/secret/fb84233d-25e0-48bc-8899-c3a3cb3cfeb9-tls-certs\") pod \"fb84233d-25e0-48bc-8899-c3a3cb3cfeb9\" (UID: \"fb84233d-25e0-48bc-8899-c3a3cb3cfeb9\") "
Oct 30 22:23:45.753187 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:45.753022    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb84233d-25e0-48bc-8899-c3a3cb3cfeb9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb84233d-25e0-48bc-8899-c3a3cb3cfeb9-tmp-dir\") pod \"fb84233d-25e0-48bc-8899-c3a3cb3cfeb9\" (UID: \"fb84233d-25e0-48bc-8899-c3a3cb3cfeb9\") "
Oct 30 22:23:45.753187 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:45.753058    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb84233d-25e0-48bc-8899-c3a3cb3cfeb9-home\" (UniqueName: \"kubernetes.io/empty-dir/fb84233d-25e0-48bc-8899-c3a3cb3cfeb9-home\") pod \"fb84233d-25e0-48bc-8899-c3a3cb3cfeb9\" (UID: \"fb84233d-25e0-48bc-8899-c3a3cb3cfeb9\") "
Oct 30 22:23:45.753187 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:45.753112    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44b7d6f6-32d8-4ef4-95bc-eef604fa6b5b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/44b7d6f6-32d8-4ef4-95bc-eef604fa6b5b-dshm\") pod \"44b7d6f6-32d8-4ef4-95bc-eef604fa6b5b\" (UID: \"44b7d6f6-32d8-4ef4-95bc-eef604fa6b5b\") "
Oct 30 22:23:45.753187 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:45.753167    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb84233d-25e0-48bc-8899-c3a3cb3cfeb9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb84233d-25e0-48bc-8899-c3a3cb3cfeb9-dshm\") pod \"fb84233d-25e0-48bc-8899-c3a3cb3cfeb9\" (UID: \"fb84233d-25e0-48bc-8899-c3a3cb3cfeb9\") "
Oct 30 22:23:45.753383 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:45.753177    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44b7d6f6-32d8-4ef4-95bc-eef604fa6b5b-tmp-dir" pod "44b7d6f6-32d8-4ef4-95bc-eef604fa6b5b" (UID: "44b7d6f6-32d8-4ef4-95bc-eef604fa6b5b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:23:45.753660 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:45.753642    5061 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44b7d6f6-32d8-4ef4-95bc-eef604fa6b5b-model-cache\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:45.753771 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:45.753664    5061 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44b7d6f6-32d8-4ef4-95bc-eef604fa6b5b-tmp-dir\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:45.754149 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:45.754122    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb84233d-25e0-48bc-8899-c3a3cb3cfeb9-model-cache" pod "fb84233d-25e0-48bc-8899-c3a3cb3cfeb9" (UID: "fb84233d-25e0-48bc-8899-c3a3cb3cfeb9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:23:45.754149 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:45.754135    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb84233d-25e0-48bc-8899-c3a3cb3cfeb9-tmp-dir" pod "fb84233d-25e0-48bc-8899-c3a3cb3cfeb9" (UID: "fb84233d-25e0-48bc-8899-c3a3cb3cfeb9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:23:45.754376 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:45.754354    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44b7d6f6-32d8-4ef4-95bc-eef604fa6b5b-home" pod "44b7d6f6-32d8-4ef4-95bc-eef604fa6b5b" (UID: "44b7d6f6-32d8-4ef4-95bc-eef604fa6b5b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:23:45.754466 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:45.754425    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb84233d-25e0-48bc-8899-c3a3cb3cfeb9-home" pod "fb84233d-25e0-48bc-8899-c3a3cb3cfeb9" (UID: "fb84233d-25e0-48bc-8899-c3a3cb3cfeb9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:23:45.755578 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:45.755515    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb84233d-25e0-48bc-8899-c3a3cb3cfeb9-tls-certs" pod "fb84233d-25e0-48bc-8899-c3a3cb3cfeb9" (UID: "fb84233d-25e0-48bc-8899-c3a3cb3cfeb9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:23:45.755578 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:45.755519    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44b7d6f6-32d8-4ef4-95bc-eef604fa6b5b-kube-api-access-pg9wx" pod "44b7d6f6-32d8-4ef4-95bc-eef604fa6b5b" (UID: "44b7d6f6-32d8-4ef4-95bc-eef604fa6b5b"). InnerVolumeSpecName "kube-api-access-pg9wx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:23:45.756622 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:45.756595    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44b7d6f6-32d8-4ef4-95bc-eef604fa6b5b-tls-certs" pod "44b7d6f6-32d8-4ef4-95bc-eef604fa6b5b" (UID: "44b7d6f6-32d8-4ef4-95bc-eef604fa6b5b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:23:45.756976 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:45.756953    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb84233d-25e0-48bc-8899-c3a3cb3cfeb9-dshm" pod "fb84233d-25e0-48bc-8899-c3a3cb3cfeb9" (UID: "fb84233d-25e0-48bc-8899-c3a3cb3cfeb9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:23:45.757334 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:45.757314    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb84233d-25e0-48bc-8899-c3a3cb3cfeb9-kube-api-access-8z9h2" pod "fb84233d-25e0-48bc-8899-c3a3cb3cfeb9" (UID: "fb84233d-25e0-48bc-8899-c3a3cb3cfeb9"). InnerVolumeSpecName "kube-api-access-8z9h2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:23:45.758415 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:45.758391    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44b7d6f6-32d8-4ef4-95bc-eef604fa6b5b-dshm" pod "44b7d6f6-32d8-4ef4-95bc-eef604fa6b5b" (UID: "44b7d6f6-32d8-4ef4-95bc-eef604fa6b5b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:23:45.855252 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:45.855221    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8z9h2\" (UniqueName: \"kubernetes.io/projected/fb84233d-25e0-48bc-8899-c3a3cb3cfeb9-kube-api-access-8z9h2\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:45.855252 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:45.855251    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pg9wx\" (UniqueName: \"kubernetes.io/projected/44b7d6f6-32d8-4ef4-95bc-eef604fa6b5b-kube-api-access-pg9wx\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:45.855507 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:45.855266    5061 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb84233d-25e0-48bc-8899-c3a3cb3cfeb9-model-cache\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:45.855507 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:45.855278    5061 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44b7d6f6-32d8-4ef4-95bc-eef604fa6b5b-home\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:45.855507 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:45.855290    5061 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44b7d6f6-32d8-4ef4-95bc-eef604fa6b5b-tls-certs\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:45.855507 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:45.855302    5061 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb84233d-25e0-48bc-8899-c3a3cb3cfeb9-tls-certs\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:45.855507 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:45.855328    5061 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb84233d-25e0-48bc-8899-c3a3cb3cfeb9-tmp-dir\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:45.855507 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:45.855339    5061 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb84233d-25e0-48bc-8899-c3a3cb3cfeb9-home\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:45.855507 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:45.855350    5061 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44b7d6f6-32d8-4ef4-95bc-eef604fa6b5b-dshm\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:45.855507 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:45.855360    5061 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb84233d-25e0-48bc-8899-c3a3cb3cfeb9-dshm\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:46.630985 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:46.630946    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2d1f91/simulator-w0-2d1f91-kserve-7699765c74-jwtkb" podUID="44b7d6f6-32d8-4ef4-95bc-eef604fa6b5b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:46.652492 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:46.652418    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2d1f91/premium-w0-2d1f91-kserve-6965b558cf-59lgw"
Oct 30 22:23:46.652492 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:46.652450    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2d1f91/premium-w0-2d1f91-kserve-6965b558cf-59lgw" event={"ID":"fb84233d-25e0-48bc-8899-c3a3cb3cfeb9","Type":"ContainerDied","Data":"11885bf7aecadd2497da239c388e9f46d1c23c5a61f7b906ab4d432ba6db17ca"}
Oct 30 22:23:46.652707 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:46.652496    5061 scope.go:122] "RemoveContainer" containerID="161ec0d5ec48158bd3d0eac0af1fc29538a01f4d1682c350cb8e36c822315f22"
Oct 30 22:23:46.654081 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:46.654064    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2d1f91/simulator-w0-2d1f91-kserve-7699765c74-jwtkb"
Oct 30 22:23:46.654167 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:46.654064    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2d1f91/simulator-w0-2d1f91-kserve-7699765c74-jwtkb" event={"ID":"44b7d6f6-32d8-4ef4-95bc-eef604fa6b5b","Type":"ContainerDied","Data":"3617d361abd37f8a3cae63d267bedb2705258d6559fe26ef947385b3e38b58f2"}
Oct 30 22:23:46.684981 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:46.684954    5061 scope.go:122] "RemoveContainer" containerID="5d169ae8c76d09ccb1d3892d416f889929adf8f367b589272f0a20ec1d5e5c97"
Oct 30 22:23:46.807969 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:46.807925    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-2d1f91/simulator-w0-2d1f91-kserve-7699765c74-jwtkb"]
Oct 30 22:23:46.855018 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:46.854985    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-2d1f91/simulator-w0-2d1f91-kserve-7699765c74-jwtkb"]
Oct 30 22:23:46.915202 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:46.915115    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-2d1f91/premium-w0-2d1f91-kserve-6965b558cf-59lgw"]
Oct 30 22:23:46.918809 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:46.918784    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-2d1f91/premium-w0-2d1f91-kserve-6965b558cf-59lgw"]
Oct 30 22:23:48.528418 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:48.528383    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44b7d6f6-32d8-4ef4-95bc-eef604fa6b5b" path="/var/lib/kubelet/pods/44b7d6f6-32d8-4ef4-95bc-eef604fa6b5b/volumes"
Oct 30 22:23:48.528855 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:48.528839    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb84233d-25e0-48bc-8899-c3a3cb3cfeb9" path="/var/lib/kubelet/pods/fb84233d-25e0-48bc-8899-c3a3cb3cfeb9/volumes"
Oct 30 22:23:51.150722 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:51.150689    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9563d02b-vbxjb"]
Oct 30 22:23:51.151311 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:51.151296    5061 cpu_manager.go:414] "containerMap: removing container" podUID="44b7d6f6-32d8-4ef4-95bc-eef604fa6b5b" containerName="main"
Oct 30 22:23:51.151311 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:51.151310    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44b7d6f6-32d8-4ef4-95bc-eef604fa6b5b" containerName="main"
Oct 30 22:23:51.151429 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:51.151321    5061 cpu_manager.go:414] "containerMap: removing container" podUID="fb84233d-25e0-48bc-8899-c3a3cb3cfeb9" containerName="main"
Oct 30 22:23:51.151429 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:51.151326    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb84233d-25e0-48bc-8899-c3a3cb3cfeb9" containerName="main"
Oct 30 22:23:51.151429 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:51.151336    5061 cpu_manager.go:414] "containerMap: removing container" podUID="d71ce3d8-00ba-4ff3-a870-6dd143dbaf93" containerName="payload-processing"
Oct 30 22:23:51.151429 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:51.151341    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d71ce3d8-00ba-4ff3-a870-6dd143dbaf93" containerName="payload-processing"
Oct 30 22:23:51.151685 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:51.151471    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="44b7d6f6-32d8-4ef4-95bc-eef604fa6b5b" containerName="main"
Oct 30 22:23:51.151685 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:51.151482    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb84233d-25e0-48bc-8899-c3a3cb3cfeb9" containerName="main"
Oct 30 22:23:51.151685 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:51.151489    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="d71ce3d8-00ba-4ff3-a870-6dd143dbaf93" containerName="payload-processing"
Oct 30 22:23:51.156480 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:51.156462    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9563d02b-vbxjb"
Oct 30 22:23:51.206443 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:51.206418    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c806f79-btkzb"]
Oct 30 22:23:51.210687 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:51.210668    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c806f79-btkzb"
Oct 30 22:23:51.229118 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:51.229090    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44457901-5c4b-48da-84ee-e5a5fca8b14e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9563d02b-vbxjb\" (UID: \"44457901-5c4b-48da-84ee-e5a5fca8b14e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9563d02b-vbxjb"
Oct 30 22:23:51.229240 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:51.229137    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44457901-5c4b-48da-84ee-e5a5fca8b14e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9563d02b-vbxjb\" (UID: \"44457901-5c4b-48da-84ee-e5a5fca8b14e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9563d02b-vbxjb"
Oct 30 22:23:51.229314 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:51.229285    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glkpl\" (UniqueName: \"kubernetes.io/projected/44457901-5c4b-48da-84ee-e5a5fca8b14e-kube-api-access-glkpl\") pod \"maas-api-revoke-sub-9563d02b-vbxjb\" (UID: \"44457901-5c4b-48da-84ee-e5a5fca8b14e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9563d02b-vbxjb"
Oct 30 22:23:51.254342 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:51.254317    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9563d02b-vbxjb"]
Oct 30 22:23:51.257089 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:51.257063    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:23:51.257231 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:51.257201    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gdmfg\""
Oct 30 22:23:51.272730 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:51.272708    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c806f79-btkzb"]
Oct 30 22:23:51.330486 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:51.330455    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgjsk\" (UniqueName: \"kubernetes.io/projected/00b6569b-f05f-4260-b9ea-a0dbb6f1a926-kube-api-access-hgjsk\") pod \"maas-api-revoke-sub-9c806f79-btkzb\" (UID: \"00b6569b-f05f-4260-b9ea-a0dbb6f1a926\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c806f79-btkzb"
Oct 30 22:23:51.330681 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:51.330515    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44457901-5c4b-48da-84ee-e5a5fca8b14e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9563d02b-vbxjb\" (UID: \"44457901-5c4b-48da-84ee-e5a5fca8b14e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9563d02b-vbxjb"
Oct 30 22:23:51.330681 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:51.330652    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44457901-5c4b-48da-84ee-e5a5fca8b14e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9563d02b-vbxjb\" (UID: \"44457901-5c4b-48da-84ee-e5a5fca8b14e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9563d02b-vbxjb"
Oct 30 22:23:51.330813 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:51.330744    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glkpl\" (UniqueName: \"kubernetes.io/projected/44457901-5c4b-48da-84ee-e5a5fca8b14e-kube-api-access-glkpl\") pod \"maas-api-revoke-sub-9563d02b-vbxjb\" (UID: \"44457901-5c4b-48da-84ee-e5a5fca8b14e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9563d02b-vbxjb"
Oct 30 22:23:51.330813 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:51.330768    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00b6569b-f05f-4260-b9ea-a0dbb6f1a926-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c806f79-btkzb\" (UID: \"00b6569b-f05f-4260-b9ea-a0dbb6f1a926\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c806f79-btkzb"
Oct 30 22:23:51.330813 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:51.330794    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00b6569b-f05f-4260-b9ea-a0dbb6f1a926-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c806f79-btkzb\" (UID: \"00b6569b-f05f-4260-b9ea-a0dbb6f1a926\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c806f79-btkzb"
Oct 30 22:23:51.331368 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:51.331347    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44457901-5c4b-48da-84ee-e5a5fca8b14e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9563d02b-vbxjb\" (UID: \"44457901-5c4b-48da-84ee-e5a5fca8b14e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9563d02b-vbxjb"
Oct 30 22:23:51.332995 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:51.332973    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44457901-5c4b-48da-84ee-e5a5fca8b14e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9563d02b-vbxjb\" (UID: \"44457901-5c4b-48da-84ee-e5a5fca8b14e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9563d02b-vbxjb"
Oct 30 22:23:51.385561 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:51.385514    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-glkpl\" (UniqueName: \"kubernetes.io/projected/44457901-5c4b-48da-84ee-e5a5fca8b14e-kube-api-access-glkpl\") pod \"maas-api-revoke-sub-9563d02b-vbxjb\" (UID: \"44457901-5c4b-48da-84ee-e5a5fca8b14e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9563d02b-vbxjb"
Oct 30 22:23:51.432307 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:51.432248    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00b6569b-f05f-4260-b9ea-a0dbb6f1a926-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c806f79-btkzb\" (UID: \"00b6569b-f05f-4260-b9ea-a0dbb6f1a926\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c806f79-btkzb"
Oct 30 22:23:51.432307 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:51.432284    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00b6569b-f05f-4260-b9ea-a0dbb6f1a926-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c806f79-btkzb\" (UID: \"00b6569b-f05f-4260-b9ea-a0dbb6f1a926\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c806f79-btkzb"
Oct 30 22:23:51.432482 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:51.432365    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgjsk\" (UniqueName: \"kubernetes.io/projected/00b6569b-f05f-4260-b9ea-a0dbb6f1a926-kube-api-access-hgjsk\") pod \"maas-api-revoke-sub-9c806f79-btkzb\" (UID: \"00b6569b-f05f-4260-b9ea-a0dbb6f1a926\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c806f79-btkzb"
Oct 30 22:23:51.432944 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:51.432921    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00b6569b-f05f-4260-b9ea-a0dbb6f1a926-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c806f79-btkzb\" (UID: \"00b6569b-f05f-4260-b9ea-a0dbb6f1a926\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c806f79-btkzb"
Oct 30 22:23:51.434470 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:51.434449    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00b6569b-f05f-4260-b9ea-a0dbb6f1a926-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c806f79-btkzb\" (UID: \"00b6569b-f05f-4260-b9ea-a0dbb6f1a926\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c806f79-btkzb"
Oct 30 22:23:51.469722 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:51.469702    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9563d02b-vbxjb"
Oct 30 22:23:51.486208 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:51.486182    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgjsk\" (UniqueName: \"kubernetes.io/projected/00b6569b-f05f-4260-b9ea-a0dbb6f1a926-kube-api-access-hgjsk\") pod \"maas-api-revoke-sub-9c806f79-btkzb\" (UID: \"00b6569b-f05f-4260-b9ea-a0dbb6f1a926\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c806f79-btkzb"
Oct 30 22:23:51.525103 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:51.525029    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c806f79-btkzb"
Oct 30 22:23:51.837872 ip-10-0-130-60 kubenswrapper[5061]: W1030 22:23:51.837829    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44457901_5c4b_48da_84ee_e5a5fca8b14e.slice/crio-12867a51c897492b012e4afebfae9cba871c9215918de666449feba6015ec9ed WatchSource:0}: Error finding container 12867a51c897492b012e4afebfae9cba871c9215918de666449feba6015ec9ed: Status 404 returned error can't find the container with id 12867a51c897492b012e4afebfae9cba871c9215918de666449feba6015ec9ed
Oct 30 22:23:51.838768 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:51.838708    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9563d02b-vbxjb"]
Oct 30 22:23:51.939314 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:51.939137    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c806f79-btkzb"]
Oct 30 22:23:51.939561 ip-10-0-130-60 kubenswrapper[5061]: W1030 22:23:51.939516    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00b6569b_f05f_4260_b9ea_a0dbb6f1a926.slice/crio-bc48c3e764a7f81a09b2665261d13e264840eec15ce7fce25b21bd99746c9470 WatchSource:0}: Error finding container bc48c3e764a7f81a09b2665261d13e264840eec15ce7fce25b21bd99746c9470: Status 404 returned error can't find the container with id bc48c3e764a7f81a09b2665261d13e264840eec15ce7fce25b21bd99746c9470
Oct 30 22:23:52.697098 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:52.697065    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c806f79-btkzb" event={"ID":"00b6569b-f05f-4260-b9ea-a0dbb6f1a926","Type":"ContainerStarted","Data":"acab0bdf919b22b07ef8c5dfc098f5d77577e07e0bd0e2c447feef0616d07b8a"}
Oct 30 22:23:52.697463 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:52.697125    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c806f79-btkzb" event={"ID":"00b6569b-f05f-4260-b9ea-a0dbb6f1a926","Type":"ContainerStarted","Data":"bc48c3e764a7f81a09b2665261d13e264840eec15ce7fce25b21bd99746c9470"}
Oct 30 22:23:52.698772 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:52.698748    5061 generic.go:356] "Generic (PLEG): container finished" podID="44457901-5c4b-48da-84ee-e5a5fca8b14e" containerID="8c86c3de310ba46ff80d5698622bcf654dd7deae2dc7a0a09907b50dd59d6e06" exitCode=0
Oct 30 22:23:52.698855 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:52.698782    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9563d02b-vbxjb" event={"ID":"44457901-5c4b-48da-84ee-e5a5fca8b14e","Type":"ContainerDied","Data":"8c86c3de310ba46ff80d5698622bcf654dd7deae2dc7a0a09907b50dd59d6e06"}
Oct 30 22:23:52.698855 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:52.698814    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9563d02b-vbxjb" event={"ID":"44457901-5c4b-48da-84ee-e5a5fca8b14e","Type":"ContainerStarted","Data":"12867a51c897492b012e4afebfae9cba871c9215918de666449feba6015ec9ed"}
Oct 30 22:23:52.809492 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:52.809452    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c806f79-btkzb" podStartSLOduration=1.8094377270000002 podStartE2EDuration="1.809437727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:23:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:23:52.808379129 +0000 UTC m=+2486.421148889" watchObservedRunningTime="2026-10-30 22:23:52.809437727 +0000 UTC m=+2486.422207484"
Oct 30 22:23:53.706665 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:53.706635    5061 generic.go:356] "Generic (PLEG): container finished" podID="00b6569b-f05f-4260-b9ea-a0dbb6f1a926" containerID="acab0bdf919b22b07ef8c5dfc098f5d77577e07e0bd0e2c447feef0616d07b8a" exitCode=0
Oct 30 22:23:53.707109 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:53.706727    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c806f79-btkzb" event={"ID":"00b6569b-f05f-4260-b9ea-a0dbb6f1a926","Type":"ContainerDied","Data":"acab0bdf919b22b07ef8c5dfc098f5d77577e07e0bd0e2c447feef0616d07b8a"}
Oct 30 22:23:53.879827 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:53.879796    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9563d02b-vbxjb"
Oct 30 22:23:53.962767 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:53.962684    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/44457901-5c4b-48da-84ee-e5a5fca8b14e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44457901-5c4b-48da-84ee-e5a5fca8b14e-maas-api-xks-ca\") pod \"44457901-5c4b-48da-84ee-e5a5fca8b14e\" (UID: \"44457901-5c4b-48da-84ee-e5a5fca8b14e\") "
Oct 30 22:23:53.962944 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:53.962781    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44457901-5c4b-48da-84ee-e5a5fca8b14e-kube-api-access-glkpl\" (UniqueName: \"kubernetes.io/projected/44457901-5c4b-48da-84ee-e5a5fca8b14e-kube-api-access-glkpl\") pod \"44457901-5c4b-48da-84ee-e5a5fca8b14e\" (UID: \"44457901-5c4b-48da-84ee-e5a5fca8b14e\") "
Oct 30 22:23:53.962944 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:53.962892    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44457901-5c4b-48da-84ee-e5a5fca8b14e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44457901-5c4b-48da-84ee-e5a5fca8b14e-maas-api-openshift-ca\") pod \"44457901-5c4b-48da-84ee-e5a5fca8b14e\" (UID: \"44457901-5c4b-48da-84ee-e5a5fca8b14e\") "
Oct 30 22:23:53.963258 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:53.963226    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44457901-5c4b-48da-84ee-e5a5fca8b14e-maas-api-openshift-ca" pod "44457901-5c4b-48da-84ee-e5a5fca8b14e" (UID: "44457901-5c4b-48da-84ee-e5a5fca8b14e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:23:53.963480 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:53.963462    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44457901-5c4b-48da-84ee-e5a5fca8b14e-maas-api-openshift-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:53.964917 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:53.964893    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44457901-5c4b-48da-84ee-e5a5fca8b14e-maas-api-xks-ca" pod "44457901-5c4b-48da-84ee-e5a5fca8b14e" (UID: "44457901-5c4b-48da-84ee-e5a5fca8b14e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:23:53.965017 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:53.964984    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44457901-5c4b-48da-84ee-e5a5fca8b14e-kube-api-access-glkpl" pod "44457901-5c4b-48da-84ee-e5a5fca8b14e" (UID: "44457901-5c4b-48da-84ee-e5a5fca8b14e"). InnerVolumeSpecName "kube-api-access-glkpl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:23:54.064959 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:54.064922    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44457901-5c4b-48da-84ee-e5a5fca8b14e-maas-api-xks-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:54.064959 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:54.064954    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glkpl\" (UniqueName: \"kubernetes.io/projected/44457901-5c4b-48da-84ee-e5a5fca8b14e-kube-api-access-glkpl\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:54.666982 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:23:54.666945    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 22:23:54.714440 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:54.714406    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9563d02b-vbxjb"
Oct 30 22:23:54.714440 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:54.714433    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9563d02b-vbxjb" event={"ID":"44457901-5c4b-48da-84ee-e5a5fca8b14e","Type":"ContainerDied","Data":"12867a51c897492b012e4afebfae9cba871c9215918de666449feba6015ec9ed"}
Oct 30 22:23:54.714913 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:54.714478    5061 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="12867a51c897492b012e4afebfae9cba871c9215918de666449feba6015ec9ed"
Oct 30 22:23:54.854939 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:54.854917    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c806f79-btkzb"
Oct 30 22:23:54.976104 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:54.976072    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00b6569b-f05f-4260-b9ea-a0dbb6f1a926-kube-api-access-hgjsk\" (UniqueName: \"kubernetes.io/projected/00b6569b-f05f-4260-b9ea-a0dbb6f1a926-kube-api-access-hgjsk\") pod \"00b6569b-f05f-4260-b9ea-a0dbb6f1a926\" (UID: \"00b6569b-f05f-4260-b9ea-a0dbb6f1a926\") "
Oct 30 22:23:54.976290 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:54.976159    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00b6569b-f05f-4260-b9ea-a0dbb6f1a926-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00b6569b-f05f-4260-b9ea-a0dbb6f1a926-maas-api-openshift-ca\") pod \"00b6569b-f05f-4260-b9ea-a0dbb6f1a926\" (UID: \"00b6569b-f05f-4260-b9ea-a0dbb6f1a926\") "
Oct 30 22:23:54.976366 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:54.976332    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00b6569b-f05f-4260-b9ea-a0dbb6f1a926-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00b6569b-f05f-4260-b9ea-a0dbb6f1a926-maas-api-xks-ca\") pod \"00b6569b-f05f-4260-b9ea-a0dbb6f1a926\" (UID: \"00b6569b-f05f-4260-b9ea-a0dbb6f1a926\") "
Oct 30 22:23:54.976563 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:54.976503    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00b6569b-f05f-4260-b9ea-a0dbb6f1a926-maas-api-openshift-ca" pod "00b6569b-f05f-4260-b9ea-a0dbb6f1a926" (UID: "00b6569b-f05f-4260-b9ea-a0dbb6f1a926"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:23:54.976879 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:54.976854    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00b6569b-f05f-4260-b9ea-a0dbb6f1a926-maas-api-openshift-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:54.978772 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:54.978750    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00b6569b-f05f-4260-b9ea-a0dbb6f1a926-maas-api-xks-ca" pod "00b6569b-f05f-4260-b9ea-a0dbb6f1a926" (UID: "00b6569b-f05f-4260-b9ea-a0dbb6f1a926"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:23:54.978895 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:54.978799    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00b6569b-f05f-4260-b9ea-a0dbb6f1a926-kube-api-access-hgjsk" pod "00b6569b-f05f-4260-b9ea-a0dbb6f1a926" (UID: "00b6569b-f05f-4260-b9ea-a0dbb6f1a926"). InnerVolumeSpecName "kube-api-access-hgjsk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:23:55.078442 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:55.078370    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00b6569b-f05f-4260-b9ea-a0dbb6f1a926-maas-api-xks-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:55.078442 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:55.078395    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgjsk\" (UniqueName: \"kubernetes.io/projected/00b6569b-f05f-4260-b9ea-a0dbb6f1a926-kube-api-access-hgjsk\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:55.722150 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:55.722115    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c806f79-btkzb" event={"ID":"00b6569b-f05f-4260-b9ea-a0dbb6f1a926","Type":"ContainerDied","Data":"bc48c3e764a7f81a09b2665261d13e264840eec15ce7fce25b21bd99746c9470"}
Oct 30 22:23:55.722559 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:55.722163    5061 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bc48c3e764a7f81a09b2665261d13e264840eec15ce7fce25b21bd99746c9470"
Oct 30 22:23:55.722559 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:23:55.722126    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c806f79-btkzb"
Oct 30 22:24:12.102499 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:12.102458    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91b15223-zw69h"]
Oct 30 22:24:12.151336 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:12.151267    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91b15223-zw69h"]
Oct 30 22:24:12.209016 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:12.208981    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-3aa85e/test-model-3aa85e-kserve-7dc5f8f957-99956"]
Oct 30 22:24:12.210142 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:12.210114    5061 cpu_manager.go:414] "containerMap: removing container" podUID="44457901-5c4b-48da-84ee-e5a5fca8b14e" containerName="revoke-keys"
Oct 30 22:24:12.210312 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:12.210297    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44457901-5c4b-48da-84ee-e5a5fca8b14e" containerName="revoke-keys"
Oct 30 22:24:12.210452 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:12.210440    5061 cpu_manager.go:414] "containerMap: removing container" podUID="00b6569b-f05f-4260-b9ea-a0dbb6f1a926" containerName="revoke-keys"
Oct 30 22:24:12.210565 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:12.210551    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00b6569b-f05f-4260-b9ea-a0dbb6f1a926" containerName="revoke-keys"
Oct 30 22:24:12.210846 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:12.210829    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="44457901-5c4b-48da-84ee-e5a5fca8b14e" containerName="revoke-keys"
Oct 30 22:24:12.210970 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:12.210959    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="00b6569b-f05f-4260-b9ea-a0dbb6f1a926" containerName="revoke-keys"
Oct 30 22:24:12.216408 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:12.216385    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-3aa85e/test-model-3aa85e-kserve-7dc5f8f957-99956"
Oct 30 22:24:12.234765 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:12.234735    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-3aa85e/test-model-3aa85e-kserve-7dc5f8f957-99956"]
Oct 30 22:24:12.267576 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:12.267546    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-3aa85e\"/\"openshift-service-ca.crt\""
Oct 30 22:24:12.268032 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:12.268016    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-3aa85e\"/\"kube-root-ca.crt\""
Oct 30 22:24:12.272379 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:12.272355    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-3aa85e\"/\"default-dockercfg-hc85r\""
Oct 30 22:24:12.272476 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:12.272435    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-3aa85e\"/\"test-model-3aa85e-kserve-self-signed-certs\""
Oct 30 22:24:12.288428 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:12.288393    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18626f54-af52-42d0-a879-be301c98a500-model-cache\") pod \"test-model-3aa85e-kserve-7dc5f8f957-99956\" (UID: \"18626f54-af52-42d0-a879-be301c98a500\") " pod="ai-tenant-e2e-inf-a-3aa85e/test-model-3aa85e-kserve-7dc5f8f957-99956"
Oct 30 22:24:12.288650 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:12.288629    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brg5m\" (UniqueName: \"kubernetes.io/projected/18626f54-af52-42d0-a879-be301c98a500-kube-api-access-brg5m\") pod \"test-model-3aa85e-kserve-7dc5f8f957-99956\" (UID: \"18626f54-af52-42d0-a879-be301c98a500\") " pod="ai-tenant-e2e-inf-a-3aa85e/test-model-3aa85e-kserve-7dc5f8f957-99956"
Oct 30 22:24:12.288730 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:12.288669    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18626f54-af52-42d0-a879-be301c98a500-tls-certs\") pod \"test-model-3aa85e-kserve-7dc5f8f957-99956\" (UID: \"18626f54-af52-42d0-a879-be301c98a500\") " pod="ai-tenant-e2e-inf-a-3aa85e/test-model-3aa85e-kserve-7dc5f8f957-99956"
Oct 30 22:24:12.288730 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:12.288700    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18626f54-af52-42d0-a879-be301c98a500-dshm\") pod \"test-model-3aa85e-kserve-7dc5f8f957-99956\" (UID: \"18626f54-af52-42d0-a879-be301c98a500\") " pod="ai-tenant-e2e-inf-a-3aa85e/test-model-3aa85e-kserve-7dc5f8f957-99956"
Oct 30 22:24:12.288815 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:12.288793    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18626f54-af52-42d0-a879-be301c98a500-tmp-dir\") pod \"test-model-3aa85e-kserve-7dc5f8f957-99956\" (UID: \"18626f54-af52-42d0-a879-be301c98a500\") " pod="ai-tenant-e2e-inf-a-3aa85e/test-model-3aa85e-kserve-7dc5f8f957-99956"
Oct 30 22:24:12.288861 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:12.288826    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18626f54-af52-42d0-a879-be301c98a500-home\") pod \"test-model-3aa85e-kserve-7dc5f8f957-99956\" (UID: \"18626f54-af52-42d0-a879-be301c98a500\") " pod="ai-tenant-e2e-inf-a-3aa85e/test-model-3aa85e-kserve-7dc5f8f957-99956"
Oct 30 22:24:12.390573 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:12.390479    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brg5m\" (UniqueName: \"kubernetes.io/projected/18626f54-af52-42d0-a879-be301c98a500-kube-api-access-brg5m\") pod \"test-model-3aa85e-kserve-7dc5f8f957-99956\" (UID: \"18626f54-af52-42d0-a879-be301c98a500\") " pod="ai-tenant-e2e-inf-a-3aa85e/test-model-3aa85e-kserve-7dc5f8f957-99956"
Oct 30 22:24:12.390573 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:12.390518    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18626f54-af52-42d0-a879-be301c98a500-tls-certs\") pod \"test-model-3aa85e-kserve-7dc5f8f957-99956\" (UID: \"18626f54-af52-42d0-a879-be301c98a500\") " pod="ai-tenant-e2e-inf-a-3aa85e/test-model-3aa85e-kserve-7dc5f8f957-99956"
Oct 30 22:24:12.390831 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:12.390578    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18626f54-af52-42d0-a879-be301c98a500-dshm\") pod \"test-model-3aa85e-kserve-7dc5f8f957-99956\" (UID: \"18626f54-af52-42d0-a879-be301c98a500\") " pod="ai-tenant-e2e-inf-a-3aa85e/test-model-3aa85e-kserve-7dc5f8f957-99956"
Oct 30 22:24:12.390831 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:12.390605    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18626f54-af52-42d0-a879-be301c98a500-tmp-dir\") pod \"test-model-3aa85e-kserve-7dc5f8f957-99956\" (UID: \"18626f54-af52-42d0-a879-be301c98a500\") " pod="ai-tenant-e2e-inf-a-3aa85e/test-model-3aa85e-kserve-7dc5f8f957-99956"
Oct 30 22:24:12.390831 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:12.390630    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18626f54-af52-42d0-a879-be301c98a500-home\") pod \"test-model-3aa85e-kserve-7dc5f8f957-99956\" (UID: \"18626f54-af52-42d0-a879-be301c98a500\") " pod="ai-tenant-e2e-inf-a-3aa85e/test-model-3aa85e-kserve-7dc5f8f957-99956"
Oct 30 22:24:12.390831 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:12.390672    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18626f54-af52-42d0-a879-be301c98a500-model-cache\") pod \"test-model-3aa85e-kserve-7dc5f8f957-99956\" (UID: \"18626f54-af52-42d0-a879-be301c98a500\") " pod="ai-tenant-e2e-inf-a-3aa85e/test-model-3aa85e-kserve-7dc5f8f957-99956"
Oct 30 22:24:12.391224 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:12.391189    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18626f54-af52-42d0-a879-be301c98a500-tmp-dir\") pod \"test-model-3aa85e-kserve-7dc5f8f957-99956\" (UID: \"18626f54-af52-42d0-a879-be301c98a500\") " pod="ai-tenant-e2e-inf-a-3aa85e/test-model-3aa85e-kserve-7dc5f8f957-99956"
Oct 30 22:24:12.391315 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:12.391197    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18626f54-af52-42d0-a879-be301c98a500-model-cache\") pod \"test-model-3aa85e-kserve-7dc5f8f957-99956\" (UID: \"18626f54-af52-42d0-a879-be301c98a500\") " pod="ai-tenant-e2e-inf-a-3aa85e/test-model-3aa85e-kserve-7dc5f8f957-99956"
Oct 30 22:24:12.391315 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:12.391247    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18626f54-af52-42d0-a879-be301c98a500-home\") pod \"test-model-3aa85e-kserve-7dc5f8f957-99956\" (UID: \"18626f54-af52-42d0-a879-be301c98a500\") " pod="ai-tenant-e2e-inf-a-3aa85e/test-model-3aa85e-kserve-7dc5f8f957-99956"
Oct 30 22:24:12.393003 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:12.392983    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18626f54-af52-42d0-a879-be301c98a500-dshm\") pod \"test-model-3aa85e-kserve-7dc5f8f957-99956\" (UID: \"18626f54-af52-42d0-a879-be301c98a500\") " pod="ai-tenant-e2e-inf-a-3aa85e/test-model-3aa85e-kserve-7dc5f8f957-99956"
Oct 30 22:24:12.393263 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:12.393244    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18626f54-af52-42d0-a879-be301c98a500-tls-certs\") pod \"test-model-3aa85e-kserve-7dc5f8f957-99956\" (UID: \"18626f54-af52-42d0-a879-be301c98a500\") " pod="ai-tenant-e2e-inf-a-3aa85e/test-model-3aa85e-kserve-7dc5f8f957-99956"
Oct 30 22:24:12.445001 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:12.444966    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brg5m\" (UniqueName: \"kubernetes.io/projected/18626f54-af52-42d0-a879-be301c98a500-kube-api-access-brg5m\") pod \"test-model-3aa85e-kserve-7dc5f8f957-99956\" (UID: \"18626f54-af52-42d0-a879-be301c98a500\") " pod="ai-tenant-e2e-inf-a-3aa85e/test-model-3aa85e-kserve-7dc5f8f957-99956"
Oct 30 22:24:12.528318 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:12.528286    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="305de000-e58b-46d2-b39e-3b7c3abd4bc3" path="/var/lib/kubelet/pods/305de000-e58b-46d2-b39e-3b7c3abd4bc3/volumes"
Oct 30 22:24:12.542905 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:12.542879    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-3aa85e/test-model-3aa85e-kserve-7dc5f8f957-99956"
Oct 30 22:24:12.991433 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:12.991400    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-3aa85e/test-model-3aa85e-kserve-7dc5f8f957-99956"]
Oct 30 22:24:12.991845 ip-10-0-130-60 kubenswrapper[5061]: W1030 22:24:12.991822    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod18626f54_af52_42d0_a879_be301c98a500.slice/crio-dfd587781b01649c2bfba9cda3a091049ef5800723e2f60e22241d78d3afe35e WatchSource:0}: Error finding container dfd587781b01649c2bfba9cda3a091049ef5800723e2f60e22241d78d3afe35e: Status 404 returned error can't find the container with id dfd587781b01649c2bfba9cda3a091049ef5800723e2f60e22241d78d3afe35e
Oct 30 22:24:13.554468 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:13.554423    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a07620-7m6w7"]
Oct 30 22:24:13.601155 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:13.601118    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a07620-7m6w7"]
Oct 30 22:24:13.852058 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:13.851964    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-3aa85e/test-model-3aa85e-kserve-7dc5f8f957-99956" event={"ID":"18626f54-af52-42d0-a879-be301c98a500","Type":"ContainerStarted","Data":"fbd2aa98fdee13aad0e19acd34b0994c242bba89148a091fbfaa1c0c75b93fa2"}
Oct 30 22:24:13.852058 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:13.852016    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-3aa85e/test-model-3aa85e-kserve-7dc5f8f957-99956" event={"ID":"18626f54-af52-42d0-a879-be301c98a500","Type":"ContainerStarted","Data":"dfd587781b01649c2bfba9cda3a091049ef5800723e2f60e22241d78d3afe35e"}
Oct 30 22:24:13.966633 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:13.966576    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-3aa85e/test-model-3aa85e-kserve-7dc5f8f957-99956" podStartSLOduration=1.9665626 podStartE2EDuration="1.9665626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:24:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:24:13.964955556 +0000 UTC m=+2507.577725313" watchObservedRunningTime="2026-10-30 22:24:13.9665626 +0000 UTC m=+2507.579332358"
Oct 30 22:24:14.529386 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:14.529347    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="848948cb-8b8b-4dc7-891d-14b18e9143d8" path="/var/lib/kubelet/pods/848948cb-8b8b-4dc7-891d-14b18e9143d8/volumes"
Oct 30 22:24:14.543301 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:14.543276    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-3aa85e/test-model-3aa85e-kserve-7dc5f8f957-99956"
Oct 30 22:24:16.304963 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:16.304926    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-84a3cf/test-model-84a3cf-kserve-57554f46fc-j6rcb"]
Oct 30 22:24:16.309509 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:16.309489    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-84a3cf/test-model-84a3cf-kserve-57554f46fc-j6rcb"
Oct 30 22:24:16.348932 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:16.348897    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ca488a2-b652-49ff-a20a-17261a16a2bd-home\") pod \"test-model-84a3cf-kserve-57554f46fc-j6rcb\" (UID: \"8ca488a2-b652-49ff-a20a-17261a16a2bd\") " pod="ai-tenant-e2e-inf-b-84a3cf/test-model-84a3cf-kserve-57554f46fc-j6rcb"
Oct 30 22:24:16.349103 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:16.348948    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ca488a2-b652-49ff-a20a-17261a16a2bd-tls-certs\") pod \"test-model-84a3cf-kserve-57554f46fc-j6rcb\" (UID: \"8ca488a2-b652-49ff-a20a-17261a16a2bd\") " pod="ai-tenant-e2e-inf-b-84a3cf/test-model-84a3cf-kserve-57554f46fc-j6rcb"
Oct 30 22:24:16.349103 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:16.349087    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ca488a2-b652-49ff-a20a-17261a16a2bd-dshm\") pod \"test-model-84a3cf-kserve-57554f46fc-j6rcb\" (UID: \"8ca488a2-b652-49ff-a20a-17261a16a2bd\") " pod="ai-tenant-e2e-inf-b-84a3cf/test-model-84a3cf-kserve-57554f46fc-j6rcb"
Oct 30 22:24:16.349231 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:16.349112    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ca488a2-b652-49ff-a20a-17261a16a2bd-tmp-dir\") pod \"test-model-84a3cf-kserve-57554f46fc-j6rcb\" (UID: \"8ca488a2-b652-49ff-a20a-17261a16a2bd\") " pod="ai-tenant-e2e-inf-b-84a3cf/test-model-84a3cf-kserve-57554f46fc-j6rcb"
Oct 30 22:24:16.349231 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:16.349134    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xm5dg\" (UniqueName: \"kubernetes.io/projected/8ca488a2-b652-49ff-a20a-17261a16a2bd-kube-api-access-xm5dg\") pod \"test-model-84a3cf-kserve-57554f46fc-j6rcb\" (UID: \"8ca488a2-b652-49ff-a20a-17261a16a2bd\") " pod="ai-tenant-e2e-inf-b-84a3cf/test-model-84a3cf-kserve-57554f46fc-j6rcb"
Oct 30 22:24:16.349346 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:16.349242    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ca488a2-b652-49ff-a20a-17261a16a2bd-model-cache\") pod \"test-model-84a3cf-kserve-57554f46fc-j6rcb\" (UID: \"8ca488a2-b652-49ff-a20a-17261a16a2bd\") " pod="ai-tenant-e2e-inf-b-84a3cf/test-model-84a3cf-kserve-57554f46fc-j6rcb"
Oct 30 22:24:16.359215 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:16.359192    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-84a3cf\"/\"openshift-service-ca.crt\""
Oct 30 22:24:16.406434 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:16.406395    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-84a3cf\"/\"kube-root-ca.crt\""
Oct 30 22:24:16.406698 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:16.406546    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-84a3cf\"/\"test-model-84a3cf-kserve-self-signed-certs\""
Oct 30 22:24:16.406814 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:16.406788    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-84a3cf\"/\"default-dockercfg-nb5l5\""
Oct 30 22:24:16.410470 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:16.410444    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-84a3cf/test-model-84a3cf-kserve-57554f46fc-j6rcb"]
Oct 30 22:24:16.455184 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:16.452398    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ca488a2-b652-49ff-a20a-17261a16a2bd-dshm\") pod \"test-model-84a3cf-kserve-57554f46fc-j6rcb\" (UID: \"8ca488a2-b652-49ff-a20a-17261a16a2bd\") " pod="ai-tenant-e2e-inf-b-84a3cf/test-model-84a3cf-kserve-57554f46fc-j6rcb"
Oct 30 22:24:16.455184 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:16.452458    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ca488a2-b652-49ff-a20a-17261a16a2bd-tmp-dir\") pod \"test-model-84a3cf-kserve-57554f46fc-j6rcb\" (UID: \"8ca488a2-b652-49ff-a20a-17261a16a2bd\") " pod="ai-tenant-e2e-inf-b-84a3cf/test-model-84a3cf-kserve-57554f46fc-j6rcb"
Oct 30 22:24:16.455184 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:16.453660    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ca488a2-b652-49ff-a20a-17261a16a2bd-tmp-dir\") pod \"test-model-84a3cf-kserve-57554f46fc-j6rcb\" (UID: \"8ca488a2-b652-49ff-a20a-17261a16a2bd\") " pod="ai-tenant-e2e-inf-b-84a3cf/test-model-84a3cf-kserve-57554f46fc-j6rcb"
Oct 30 22:24:16.455184 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:16.452495    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xm5dg\" (UniqueName: \"kubernetes.io/projected/8ca488a2-b652-49ff-a20a-17261a16a2bd-kube-api-access-xm5dg\") pod \"test-model-84a3cf-kserve-57554f46fc-j6rcb\" (UID: \"8ca488a2-b652-49ff-a20a-17261a16a2bd\") " pod="ai-tenant-e2e-inf-b-84a3cf/test-model-84a3cf-kserve-57554f46fc-j6rcb"
Oct 30 22:24:16.455184 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:16.454106    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ca488a2-b652-49ff-a20a-17261a16a2bd-model-cache\") pod \"test-model-84a3cf-kserve-57554f46fc-j6rcb\" (UID: \"8ca488a2-b652-49ff-a20a-17261a16a2bd\") " pod="ai-tenant-e2e-inf-b-84a3cf/test-model-84a3cf-kserve-57554f46fc-j6rcb"
Oct 30 22:24:16.455184 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:16.454336    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ca488a2-b652-49ff-a20a-17261a16a2bd-home\") pod \"test-model-84a3cf-kserve-57554f46fc-j6rcb\" (UID: \"8ca488a2-b652-49ff-a20a-17261a16a2bd\") " pod="ai-tenant-e2e-inf-b-84a3cf/test-model-84a3cf-kserve-57554f46fc-j6rcb"
Oct 30 22:24:16.455184 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:16.454373    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ca488a2-b652-49ff-a20a-17261a16a2bd-tls-certs\") pod \"test-model-84a3cf-kserve-57554f46fc-j6rcb\" (UID: \"8ca488a2-b652-49ff-a20a-17261a16a2bd\") " pod="ai-tenant-e2e-inf-b-84a3cf/test-model-84a3cf-kserve-57554f46fc-j6rcb"
Oct 30 22:24:16.456321 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:16.455475    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ca488a2-b652-49ff-a20a-17261a16a2bd-model-cache\") pod \"test-model-84a3cf-kserve-57554f46fc-j6rcb\" (UID: \"8ca488a2-b652-49ff-a20a-17261a16a2bd\") " pod="ai-tenant-e2e-inf-b-84a3cf/test-model-84a3cf-kserve-57554f46fc-j6rcb"
Oct 30 22:24:16.456321 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:16.455761    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ca488a2-b652-49ff-a20a-17261a16a2bd-home\") pod \"test-model-84a3cf-kserve-57554f46fc-j6rcb\" (UID: \"8ca488a2-b652-49ff-a20a-17261a16a2bd\") " pod="ai-tenant-e2e-inf-b-84a3cf/test-model-84a3cf-kserve-57554f46fc-j6rcb"
Oct 30 22:24:16.461667 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:16.461627    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ca488a2-b652-49ff-a20a-17261a16a2bd-tls-certs\") pod \"test-model-84a3cf-kserve-57554f46fc-j6rcb\" (UID: \"8ca488a2-b652-49ff-a20a-17261a16a2bd\") " pod="ai-tenant-e2e-inf-b-84a3cf/test-model-84a3cf-kserve-57554f46fc-j6rcb"
Oct 30 22:24:16.469958 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:16.469926    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ca488a2-b652-49ff-a20a-17261a16a2bd-dshm\") pod \"test-model-84a3cf-kserve-57554f46fc-j6rcb\" (UID: \"8ca488a2-b652-49ff-a20a-17261a16a2bd\") " pod="ai-tenant-e2e-inf-b-84a3cf/test-model-84a3cf-kserve-57554f46fc-j6rcb"
Oct 30 22:24:16.507947 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:16.507915    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xm5dg\" (UniqueName: \"kubernetes.io/projected/8ca488a2-b652-49ff-a20a-17261a16a2bd-kube-api-access-xm5dg\") pod \"test-model-84a3cf-kserve-57554f46fc-j6rcb\" (UID: \"8ca488a2-b652-49ff-a20a-17261a16a2bd\") " pod="ai-tenant-e2e-inf-b-84a3cf/test-model-84a3cf-kserve-57554f46fc-j6rcb"
Oct 30 22:24:16.623193 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:16.623085    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-84a3cf/test-model-84a3cf-kserve-57554f46fc-j6rcb"
Oct 30 22:24:16.992424 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:16.992396    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-84a3cf/test-model-84a3cf-kserve-57554f46fc-j6rcb"]
Oct 30 22:24:16.992696 ip-10-0-130-60 kubenswrapper[5061]: W1030 22:24:16.992671    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8ca488a2_b652_49ff_a20a_17261a16a2bd.slice/crio-2426080d2a521c92b37932e72b7988b4b001e954ababba8ab37d1bfbd0013b21 WatchSource:0}: Error finding container 2426080d2a521c92b37932e72b7988b4b001e954ababba8ab37d1bfbd0013b21: Status 404 returned error can't find the container with id 2426080d2a521c92b37932e72b7988b4b001e954ababba8ab37d1bfbd0013b21
Oct 30 22:24:17.881639 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:17.881602    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-84a3cf/test-model-84a3cf-kserve-57554f46fc-j6rcb" event={"ID":"8ca488a2-b652-49ff-a20a-17261a16a2bd","Type":"ContainerStarted","Data":"7295e76cd512dac4ec5a11826d96c9fe5bfc2980dddea24f0bfc7252e18acec0"}
Oct 30 22:24:17.882037 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:17.881648    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-84a3cf/test-model-84a3cf-kserve-57554f46fc-j6rcb" event={"ID":"8ca488a2-b652-49ff-a20a-17261a16a2bd","Type":"ContainerStarted","Data":"2426080d2a521c92b37932e72b7988b4b001e954ababba8ab37d1bfbd0013b21"}
Oct 30 22:24:17.992522 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:17.992468    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-84a3cf/test-model-84a3cf-kserve-57554f46fc-j6rcb" podStartSLOduration=1.992455984 podStartE2EDuration="1.992455984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:24:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:24:17.990901285 +0000 UTC m=+2511.603671042" watchObservedRunningTime="2026-10-30 22:24:17.992455984 +0000 UTC m=+2511.605225794"
Oct 30 22:24:18.443125 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:18.443091    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-edf6d5-6vblg"]
Oct 30 22:24:18.447468 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:18.447444    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-edf6d5-6vblg"
Oct 30 22:24:18.500013 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:18.499982    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:24:18.500194 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:18.500176    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gdmfg\""
Oct 30 22:24:18.550879 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:18.550844    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-edf6d5-6vblg"]
Oct 30 22:24:18.584056 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:18.584022    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qx6h5\" (UniqueName: \"kubernetes.io/projected/005a7ef7-c814-4c62-9e53-21aaff9c2640-kube-api-access-qx6h5\") pod \"maas-api-revoke-keys-e2e-api-a-edf6d5-6vblg\" (UID: \"005a7ef7-c814-4c62-9e53-21aaff9c2640\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-edf6d5-6vblg"
Oct 30 22:24:18.584266 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:18.584089    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/005a7ef7-c814-4c62-9e53-21aaff9c2640-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-edf6d5-6vblg\" (UID: \"005a7ef7-c814-4c62-9e53-21aaff9c2640\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-edf6d5-6vblg"
Oct 30 22:24:18.584266 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:18.584190    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/005a7ef7-c814-4c62-9e53-21aaff9c2640-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-edf6d5-6vblg\" (UID: \"005a7ef7-c814-4c62-9e53-21aaff9c2640\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-edf6d5-6vblg"
Oct 30 22:24:18.623734 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:18.623697    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-84a3cf/test-model-84a3cf-kserve-57554f46fc-j6rcb"
Oct 30 22:24:18.685488 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:18.685443    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/005a7ef7-c814-4c62-9e53-21aaff9c2640-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-edf6d5-6vblg\" (UID: \"005a7ef7-c814-4c62-9e53-21aaff9c2640\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-edf6d5-6vblg"
Oct 30 22:24:18.685722 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:18.685512    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/005a7ef7-c814-4c62-9e53-21aaff9c2640-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-edf6d5-6vblg\" (UID: \"005a7ef7-c814-4c62-9e53-21aaff9c2640\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-edf6d5-6vblg"
Oct 30 22:24:18.685722 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:18.685665    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qx6h5\" (UniqueName: \"kubernetes.io/projected/005a7ef7-c814-4c62-9e53-21aaff9c2640-kube-api-access-qx6h5\") pod \"maas-api-revoke-keys-e2e-api-a-edf6d5-6vblg\" (UID: \"005a7ef7-c814-4c62-9e53-21aaff9c2640\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-edf6d5-6vblg"
Oct 30 22:24:18.686255 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:18.686228    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/005a7ef7-c814-4c62-9e53-21aaff9c2640-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-edf6d5-6vblg\" (UID: \"005a7ef7-c814-4c62-9e53-21aaff9c2640\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-edf6d5-6vblg"
Oct 30 22:24:18.687896 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:18.687876    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/005a7ef7-c814-4c62-9e53-21aaff9c2640-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-edf6d5-6vblg\" (UID: \"005a7ef7-c814-4c62-9e53-21aaff9c2640\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-edf6d5-6vblg"
Oct 30 22:24:18.740318 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:18.740246    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qx6h5\" (UniqueName: \"kubernetes.io/projected/005a7ef7-c814-4c62-9e53-21aaff9c2640-kube-api-access-qx6h5\") pod \"maas-api-revoke-keys-e2e-api-a-edf6d5-6vblg\" (UID: \"005a7ef7-c814-4c62-9e53-21aaff9c2640\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-edf6d5-6vblg"
Oct 30 22:24:18.762373 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:18.762346    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-edf6d5-6vblg"
Oct 30 22:24:19.129939 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:19.129877    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-edf6d5-6vblg"]
Oct 30 22:24:19.130614 ip-10-0-130-60 kubenswrapper[5061]: W1030 22:24:19.130359    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod005a7ef7_c814_4c62_9e53_21aaff9c2640.slice/crio-3ac6ea016d1637258b03d611f28b33ebfc19a76107815d7375e24a9786c67fbb WatchSource:0}: Error finding container 3ac6ea016d1637258b03d611f28b33ebfc19a76107815d7375e24a9786c67fbb: Status 404 returned error can't find the container with id 3ac6ea016d1637258b03d611f28b33ebfc19a76107815d7375e24a9786c67fbb
Oct 30 22:24:19.897608 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:19.897561    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-edf6d5-6vblg" event={"ID":"005a7ef7-c814-4c62-9e53-21aaff9c2640","Type":"ContainerStarted","Data":"bb41e5567f2211e56326899b72524c6b58b640f09f9842461f9af2133fdb048a"}
Oct 30 22:24:19.897790 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:19.897629    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-edf6d5-6vblg" event={"ID":"005a7ef7-c814-4c62-9e53-21aaff9c2640","Type":"ContainerStarted","Data":"3ac6ea016d1637258b03d611f28b33ebfc19a76107815d7375e24a9786c67fbb"}
Oct 30 22:24:20.014031 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:20.013986    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-edf6d5-6vblg" podStartSLOduration=2.013972858 podStartE2EDuration="2.013972858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:24:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:24:20.013593307 +0000 UTC m=+2513.626363064" watchObservedRunningTime="2026-10-30 22:24:20.013972858 +0000 UTC m=+2513.626742615"
Oct 30 22:24:20.906790 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:20.906751    5061 generic.go:356] "Generic (PLEG): container finished" podID="005a7ef7-c814-4c62-9e53-21aaff9c2640" containerID="bb41e5567f2211e56326899b72524c6b58b640f09f9842461f9af2133fdb048a" exitCode=0
Oct 30 22:24:20.907255 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:20.906793    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-edf6d5-6vblg" event={"ID":"005a7ef7-c814-4c62-9e53-21aaff9c2640","Type":"ContainerDied","Data":"bb41e5567f2211e56326899b72524c6b58b640f09f9842461f9af2133fdb048a"}
Oct 30 22:24:22.076889 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:22.076862    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-edf6d5-6vblg"
Oct 30 22:24:22.243326 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:22.243250    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/005a7ef7-c814-4c62-9e53-21aaff9c2640-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/005a7ef7-c814-4c62-9e53-21aaff9c2640-maas-api-xks-ca\") pod \"005a7ef7-c814-4c62-9e53-21aaff9c2640\" (UID: \"005a7ef7-c814-4c62-9e53-21aaff9c2640\") "
Oct 30 22:24:22.243477 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:22.243383    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/005a7ef7-c814-4c62-9e53-21aaff9c2640-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/005a7ef7-c814-4c62-9e53-21aaff9c2640-maas-api-openshift-ca\") pod \"005a7ef7-c814-4c62-9e53-21aaff9c2640\" (UID: \"005a7ef7-c814-4c62-9e53-21aaff9c2640\") "
Oct 30 22:24:22.243477 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:22.243441    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/005a7ef7-c814-4c62-9e53-21aaff9c2640-kube-api-access-qx6h5\" (UniqueName: \"kubernetes.io/projected/005a7ef7-c814-4c62-9e53-21aaff9c2640-kube-api-access-qx6h5\") pod \"005a7ef7-c814-4c62-9e53-21aaff9c2640\" (UID: \"005a7ef7-c814-4c62-9e53-21aaff9c2640\") "
Oct 30 22:24:22.243797 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:22.243774    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/005a7ef7-c814-4c62-9e53-21aaff9c2640-maas-api-openshift-ca" pod "005a7ef7-c814-4c62-9e53-21aaff9c2640" (UID: "005a7ef7-c814-4c62-9e53-21aaff9c2640"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:24:22.245418 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:22.245390    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/005a7ef7-c814-4c62-9e53-21aaff9c2640-maas-api-xks-ca" pod "005a7ef7-c814-4c62-9e53-21aaff9c2640" (UID: "005a7ef7-c814-4c62-9e53-21aaff9c2640"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:24:22.245603 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:22.245579    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/005a7ef7-c814-4c62-9e53-21aaff9c2640-kube-api-access-qx6h5" pod "005a7ef7-c814-4c62-9e53-21aaff9c2640" (UID: "005a7ef7-c814-4c62-9e53-21aaff9c2640"). InnerVolumeSpecName "kube-api-access-qx6h5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:24:22.345466 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:22.345426    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qx6h5\" (UniqueName: \"kubernetes.io/projected/005a7ef7-c814-4c62-9e53-21aaff9c2640-kube-api-access-qx6h5\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:24:22.345466 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:22.345457    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/005a7ef7-c814-4c62-9e53-21aaff9c2640-maas-api-xks-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:24:22.345466 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:22.345469    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/005a7ef7-c814-4c62-9e53-21aaff9c2640-maas-api-openshift-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:24:22.543097 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:22.543068    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-3aa85e/test-model-3aa85e-kserve-7dc5f8f957-99956"
Oct 30 22:24:22.555711 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:22.555687    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-3aa85e/test-model-3aa85e-kserve-7dc5f8f957-99956"
Oct 30 22:24:22.922415 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:22.922388    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-edf6d5-6vblg"
Oct 30 22:24:22.922653 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:22.922427    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-edf6d5-6vblg" event={"ID":"005a7ef7-c814-4c62-9e53-21aaff9c2640","Type":"ContainerDied","Data":"3ac6ea016d1637258b03d611f28b33ebfc19a76107815d7375e24a9786c67fbb"}
Oct 30 22:24:22.922653 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:22.922482    5061 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3ac6ea016d1637258b03d611f28b33ebfc19a76107815d7375e24a9786c67fbb"
Oct 30 22:24:22.934522 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:22.934499    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-3aa85e/test-model-3aa85e-kserve-7dc5f8f957-99956"
Oct 30 22:24:22.941406 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:22.941382    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5632f8c-s797h"]
Oct 30 22:24:22.942084 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:22.942064    5061 cpu_manager.go:414] "containerMap: removing container" podUID="005a7ef7-c814-4c62-9e53-21aaff9c2640" containerName="revoke-keys"
Oct 30 22:24:22.942211 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:22.942085    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="005a7ef7-c814-4c62-9e53-21aaff9c2640" containerName="revoke-keys"
Oct 30 22:24:22.942683 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:22.942335    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="005a7ef7-c814-4c62-9e53-21aaff9c2640" containerName="revoke-keys"
Oct 30 22:24:22.946204 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:22.946185    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5632f8c-s797h"
Oct 30 22:24:22.998341 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:22.998306    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:24:22.998586 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:22.998356    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gdmfg\""
Oct 30 22:24:23.002510 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:23.002481    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4cbc3bf9-s8995"]
Oct 30 22:24:23.010330 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:23.010308    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cbc3bf9-s8995"
Oct 30 22:24:23.028695 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:23.028667    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5632f8c-s797h"]
Oct 30 22:24:23.054439 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:23.054416    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc06e65c-d30e-4c21-9ee5-beb7e2e1c4de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5632f8c-s797h\" (UID: \"fc06e65c-d30e-4c21-9ee5-beb7e2e1c4de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5632f8c-s797h"
Oct 30 22:24:23.054554 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:23.054454    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8s94\" (UniqueName: \"kubernetes.io/projected/fc06e65c-d30e-4c21-9ee5-beb7e2e1c4de-kube-api-access-c8s94\") pod \"maas-api-revoke-sub-c5632f8c-s797h\" (UID: \"fc06e65c-d30e-4c21-9ee5-beb7e2e1c4de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5632f8c-s797h"
Oct 30 22:24:23.054554 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:23.054478    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc06e65c-d30e-4c21-9ee5-beb7e2e1c4de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5632f8c-s797h\" (UID: \"fc06e65c-d30e-4c21-9ee5-beb7e2e1c4de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5632f8c-s797h"
Oct 30 22:24:23.068969 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:23.068942    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4cbc3bf9-s8995"]
Oct 30 22:24:23.155921 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:23.155887    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc06e65c-d30e-4c21-9ee5-beb7e2e1c4de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5632f8c-s797h\" (UID: \"fc06e65c-d30e-4c21-9ee5-beb7e2e1c4de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5632f8c-s797h"
Oct 30 22:24:23.156407 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:23.155931    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8s94\" (UniqueName: \"kubernetes.io/projected/fc06e65c-d30e-4c21-9ee5-beb7e2e1c4de-kube-api-access-c8s94\") pod \"maas-api-revoke-sub-c5632f8c-s797h\" (UID: \"fc06e65c-d30e-4c21-9ee5-beb7e2e1c4de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5632f8c-s797h"
Oct 30 22:24:23.156407 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:23.155964    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc06e65c-d30e-4c21-9ee5-beb7e2e1c4de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5632f8c-s797h\" (UID: \"fc06e65c-d30e-4c21-9ee5-beb7e2e1c4de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5632f8c-s797h"
Oct 30 22:24:23.156407 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:23.156000    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/932e2028-beda-405f-bc0a-6463a3cecac0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4cbc3bf9-s8995\" (UID: \"932e2028-beda-405f-bc0a-6463a3cecac0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cbc3bf9-s8995"
Oct 30 22:24:23.156407 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:23.156067    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2thpf\" (UniqueName: \"kubernetes.io/projected/932e2028-beda-405f-bc0a-6463a3cecac0-kube-api-access-2thpf\") pod \"maas-api-revoke-sub-4cbc3bf9-s8995\" (UID: \"932e2028-beda-405f-bc0a-6463a3cecac0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cbc3bf9-s8995"
Oct 30 22:24:23.156407 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:23.156295    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/932e2028-beda-405f-bc0a-6463a3cecac0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4cbc3bf9-s8995\" (UID: \"932e2028-beda-405f-bc0a-6463a3cecac0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cbc3bf9-s8995"
Oct 30 22:24:23.156720 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:23.156590    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc06e65c-d30e-4c21-9ee5-beb7e2e1c4de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5632f8c-s797h\" (UID: \"fc06e65c-d30e-4c21-9ee5-beb7e2e1c4de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5632f8c-s797h"
Oct 30 22:24:23.158400 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:23.158378    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc06e65c-d30e-4c21-9ee5-beb7e2e1c4de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5632f8c-s797h\" (UID: \"fc06e65c-d30e-4c21-9ee5-beb7e2e1c4de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5632f8c-s797h"
Oct 30 22:24:23.216942 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:23.216840    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8s94\" (UniqueName: \"kubernetes.io/projected/fc06e65c-d30e-4c21-9ee5-beb7e2e1c4de-kube-api-access-c8s94\") pod \"maas-api-revoke-sub-c5632f8c-s797h\" (UID: \"fc06e65c-d30e-4c21-9ee5-beb7e2e1c4de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5632f8c-s797h"
Oct 30 22:24:23.257030 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:23.257001    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2thpf\" (UniqueName: \"kubernetes.io/projected/932e2028-beda-405f-bc0a-6463a3cecac0-kube-api-access-2thpf\") pod \"maas-api-revoke-sub-4cbc3bf9-s8995\" (UID: \"932e2028-beda-405f-bc0a-6463a3cecac0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cbc3bf9-s8995"
Oct 30 22:24:23.257184 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:23.257116    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/932e2028-beda-405f-bc0a-6463a3cecac0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4cbc3bf9-s8995\" (UID: \"932e2028-beda-405f-bc0a-6463a3cecac0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cbc3bf9-s8995"
Oct 30 22:24:23.257241 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:23.257201    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/932e2028-beda-405f-bc0a-6463a3cecac0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4cbc3bf9-s8995\" (UID: \"932e2028-beda-405f-bc0a-6463a3cecac0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cbc3bf9-s8995"
Oct 30 22:24:23.257915 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:23.257890    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/932e2028-beda-405f-bc0a-6463a3cecac0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4cbc3bf9-s8995\" (UID: \"932e2028-beda-405f-bc0a-6463a3cecac0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cbc3bf9-s8995"
Oct 30 22:24:23.259482 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:23.259458    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/932e2028-beda-405f-bc0a-6463a3cecac0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4cbc3bf9-s8995\" (UID: \"932e2028-beda-405f-bc0a-6463a3cecac0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cbc3bf9-s8995"
Oct 30 22:24:23.263081 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:23.263064    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5632f8c-s797h"
Oct 30 22:24:23.313668 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:23.313641    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2thpf\" (UniqueName: \"kubernetes.io/projected/932e2028-beda-405f-bc0a-6463a3cecac0-kube-api-access-2thpf\") pod \"maas-api-revoke-sub-4cbc3bf9-s8995\" (UID: \"932e2028-beda-405f-bc0a-6463a3cecac0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cbc3bf9-s8995"
Oct 30 22:24:23.325577 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:23.324908    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cbc3bf9-s8995"
Oct 30 22:24:23.638344 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:23.638305    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5632f8c-s797h"]
Oct 30 22:24:23.638700 ip-10-0-130-60 kubenswrapper[5061]: W1030 22:24:23.638659    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc06e65c_d30e_4c21_9ee5_beb7e2e1c4de.slice/crio-c3c23cfc11136b1eeaf777dad4fce422c1493104c0fa4a4fc4ab8df8528f305e WatchSource:0}: Error finding container c3c23cfc11136b1eeaf777dad4fce422c1493104c0fa4a4fc4ab8df8528f305e: Status 404 returned error can't find the container with id c3c23cfc11136b1eeaf777dad4fce422c1493104c0fa4a4fc4ab8df8528f305e
Oct 30 22:24:23.711656 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:23.711618    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4cbc3bf9-s8995"]
Oct 30 22:24:23.931324 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:23.931243    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5632f8c-s797h" event={"ID":"fc06e65c-d30e-4c21-9ee5-beb7e2e1c4de","Type":"ContainerStarted","Data":"c3c23cfc11136b1eeaf777dad4fce422c1493104c0fa4a4fc4ab8df8528f305e"}
Oct 30 22:24:23.932599 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:23.932569    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cbc3bf9-s8995" event={"ID":"932e2028-beda-405f-bc0a-6463a3cecac0","Type":"ContainerStarted","Data":"061aaafbe9e2c858b46f8540a5d8698ab440676e0ca4eb5b402e3e998306407a"}
Oct 30 22:24:24.667103 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:24:24.667065    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 22:24:24.941595 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:24.941557    5061 generic.go:356] "Generic (PLEG): container finished" podID="932e2028-beda-405f-bc0a-6463a3cecac0" containerID="e617e63effe6985b880ad7e2ec66b5e5ee7fc768999f0d7945131ccdfa69f94e" exitCode=0
Oct 30 22:24:24.941793 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:24.941640    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cbc3bf9-s8995" event={"ID":"932e2028-beda-405f-bc0a-6463a3cecac0","Type":"ContainerDied","Data":"e617e63effe6985b880ad7e2ec66b5e5ee7fc768999f0d7945131ccdfa69f94e"}
Oct 30 22:24:24.943496 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:24.943457    5061 generic.go:356] "Generic (PLEG): container finished" podID="fc06e65c-d30e-4c21-9ee5-beb7e2e1c4de" containerID="28acd13885beb0938d1966b3106ad9025be66edda54a2a935c198c23d8d34f02" exitCode=0
Oct 30 22:24:24.943652 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:24.943550    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5632f8c-s797h" event={"ID":"fc06e65c-d30e-4c21-9ee5-beb7e2e1c4de","Type":"ContainerDied","Data":"28acd13885beb0938d1966b3106ad9025be66edda54a2a935c198c23d8d34f02"}
Oct 30 22:24:26.167093 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:26.167067    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5632f8c-s797h"
Oct 30 22:24:26.170478 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:26.170454    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cbc3bf9-s8995"
Oct 30 22:24:26.297040 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:26.297005    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc06e65c-d30e-4c21-9ee5-beb7e2e1c4de-kube-api-access-c8s94\" (UniqueName: \"kubernetes.io/projected/fc06e65c-d30e-4c21-9ee5-beb7e2e1c4de-kube-api-access-c8s94\") pod \"fc06e65c-d30e-4c21-9ee5-beb7e2e1c4de\" (UID: \"fc06e65c-d30e-4c21-9ee5-beb7e2e1c4de\") "
Oct 30 22:24:26.297246 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:26.297112    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/932e2028-beda-405f-bc0a-6463a3cecac0-kube-api-access-2thpf\" (UniqueName: \"kubernetes.io/projected/932e2028-beda-405f-bc0a-6463a3cecac0-kube-api-access-2thpf\") pod \"932e2028-beda-405f-bc0a-6463a3cecac0\" (UID: \"932e2028-beda-405f-bc0a-6463a3cecac0\") "
Oct 30 22:24:26.297246 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:26.297185    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc06e65c-d30e-4c21-9ee5-beb7e2e1c4de-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc06e65c-d30e-4c21-9ee5-beb7e2e1c4de-maas-api-openshift-ca\") pod \"fc06e65c-d30e-4c21-9ee5-beb7e2e1c4de\" (UID: \"fc06e65c-d30e-4c21-9ee5-beb7e2e1c4de\") "
Oct 30 22:24:26.297368 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:26.297278    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc06e65c-d30e-4c21-9ee5-beb7e2e1c4de-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc06e65c-d30e-4c21-9ee5-beb7e2e1c4de-maas-api-xks-ca\") pod \"fc06e65c-d30e-4c21-9ee5-beb7e2e1c4de\" (UID: \"fc06e65c-d30e-4c21-9ee5-beb7e2e1c4de\") "
Oct 30 22:24:26.297368 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:26.297318    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/932e2028-beda-405f-bc0a-6463a3cecac0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/932e2028-beda-405f-bc0a-6463a3cecac0-maas-api-xks-ca\") pod \"932e2028-beda-405f-bc0a-6463a3cecac0\" (UID: \"932e2028-beda-405f-bc0a-6463a3cecac0\") "
Oct 30 22:24:26.297478 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:26.297457    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/932e2028-beda-405f-bc0a-6463a3cecac0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/932e2028-beda-405f-bc0a-6463a3cecac0-maas-api-openshift-ca\") pod \"932e2028-beda-405f-bc0a-6463a3cecac0\" (UID: \"932e2028-beda-405f-bc0a-6463a3cecac0\") "
Oct 30 22:24:26.297666 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:26.297636    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc06e65c-d30e-4c21-9ee5-beb7e2e1c4de-maas-api-openshift-ca" pod "fc06e65c-d30e-4c21-9ee5-beb7e2e1c4de" (UID: "fc06e65c-d30e-4c21-9ee5-beb7e2e1c4de"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:24:26.298067 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:26.298011    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/932e2028-beda-405f-bc0a-6463a3cecac0-maas-api-openshift-ca" pod "932e2028-beda-405f-bc0a-6463a3cecac0" (UID: "932e2028-beda-405f-bc0a-6463a3cecac0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:24:26.298194 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:26.298073    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc06e65c-d30e-4c21-9ee5-beb7e2e1c4de-maas-api-openshift-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:24:26.299977 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:26.299954    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc06e65c-d30e-4c21-9ee5-beb7e2e1c4de-maas-api-xks-ca" pod "fc06e65c-d30e-4c21-9ee5-beb7e2e1c4de" (UID: "fc06e65c-d30e-4c21-9ee5-beb7e2e1c4de"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:24:26.300086 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:26.299979    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc06e65c-d30e-4c21-9ee5-beb7e2e1c4de-kube-api-access-c8s94" pod "fc06e65c-d30e-4c21-9ee5-beb7e2e1c4de" (UID: "fc06e65c-d30e-4c21-9ee5-beb7e2e1c4de"). InnerVolumeSpecName "kube-api-access-c8s94". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:24:26.300155 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:26.300091    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/932e2028-beda-405f-bc0a-6463a3cecac0-kube-api-access-2thpf" pod "932e2028-beda-405f-bc0a-6463a3cecac0" (UID: "932e2028-beda-405f-bc0a-6463a3cecac0"). InnerVolumeSpecName "kube-api-access-2thpf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:24:26.300707 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:26.300689    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/932e2028-beda-405f-bc0a-6463a3cecac0-maas-api-xks-ca" pod "932e2028-beda-405f-bc0a-6463a3cecac0" (UID: "932e2028-beda-405f-bc0a-6463a3cecac0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:24:26.399199 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:26.399160    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8s94\" (UniqueName: \"kubernetes.io/projected/fc06e65c-d30e-4c21-9ee5-beb7e2e1c4de-kube-api-access-c8s94\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:24:26.399199 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:26.399199    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2thpf\" (UniqueName: \"kubernetes.io/projected/932e2028-beda-405f-bc0a-6463a3cecac0-kube-api-access-2thpf\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:24:26.399464 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:26.399213    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc06e65c-d30e-4c21-9ee5-beb7e2e1c4de-maas-api-xks-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:24:26.399464 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:26.399227    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/932e2028-beda-405f-bc0a-6463a3cecac0-maas-api-xks-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:24:26.399464 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:26.399239    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/932e2028-beda-405f-bc0a-6463a3cecac0-maas-api-openshift-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:24:26.623295 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:26.623203    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-84a3cf/test-model-84a3cf-kserve-57554f46fc-j6rcb"
Oct 30 22:24:26.638695 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:26.638659    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-84a3cf/test-model-84a3cf-kserve-57554f46fc-j6rcb"
Oct 30 22:24:26.961521 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:26.961421    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5632f8c-s797h"
Oct 30 22:24:26.961521 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:26.961458    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5632f8c-s797h" event={"ID":"fc06e65c-d30e-4c21-9ee5-beb7e2e1c4de","Type":"ContainerDied","Data":"c3c23cfc11136b1eeaf777dad4fce422c1493104c0fa4a4fc4ab8df8528f305e"}
Oct 30 22:24:26.961796 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:26.961560    5061 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c3c23cfc11136b1eeaf777dad4fce422c1493104c0fa4a4fc4ab8df8528f305e"
Oct 30 22:24:26.963200 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:26.963176    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cbc3bf9-s8995"
Oct 30 22:24:26.963200 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:26.963170    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cbc3bf9-s8995" event={"ID":"932e2028-beda-405f-bc0a-6463a3cecac0","Type":"ContainerDied","Data":"061aaafbe9e2c858b46f8540a5d8698ab440676e0ca4eb5b402e3e998306407a"}
Oct 30 22:24:26.963376 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:26.963229    5061 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="061aaafbe9e2c858b46f8540a5d8698ab440676e0ca4eb5b402e3e998306407a"
Oct 30 22:24:26.978675 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:26.978647    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-84a3cf/test-model-84a3cf-kserve-57554f46fc-j6rcb"
Oct 30 22:24:28.021269 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:28.021235    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47842a96-ccbdc"]
Oct 30 22:24:28.021917 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:28.021901    5061 cpu_manager.go:414] "containerMap: removing container" podUID="932e2028-beda-405f-bc0a-6463a3cecac0" containerName="revoke-keys"
Oct 30 22:24:28.021977 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:28.021918    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="932e2028-beda-405f-bc0a-6463a3cecac0" containerName="revoke-keys"
Oct 30 22:24:28.022017 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:28.021976    5061 cpu_manager.go:414] "containerMap: removing container" podUID="fc06e65c-d30e-4c21-9ee5-beb7e2e1c4de" containerName="revoke-keys"
Oct 30 22:24:28.022017 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:28.021982    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc06e65c-d30e-4c21-9ee5-beb7e2e1c4de" containerName="revoke-keys"
Oct 30 22:24:28.022087 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:28.022077    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc06e65c-d30e-4c21-9ee5-beb7e2e1c4de" containerName="revoke-keys"
Oct 30 22:24:28.022124 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:28.022090    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="932e2028-beda-405f-bc0a-6463a3cecac0" containerName="revoke-keys"
Oct 30 22:24:28.025987 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:28.025966    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47842a96-ccbdc"
Oct 30 22:24:28.075437 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:28.075408    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:24:28.122504 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:28.122473    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gdmfg\""
Oct 30 22:24:28.126164 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:28.126142    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47842a96-ccbdc"]
Oct 30 22:24:28.221507 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:28.221472    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3c3d477-f4c2-4740-b15e-aea2e058197b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47842a96-ccbdc\" (UID: \"e3c3d477-f4c2-4740-b15e-aea2e058197b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47842a96-ccbdc"
Oct 30 22:24:28.221703 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:28.221515    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bjkf\" (UniqueName: \"kubernetes.io/projected/e3c3d477-f4c2-4740-b15e-aea2e058197b-kube-api-access-2bjkf\") pod \"maas-api-revoke-sub-47842a96-ccbdc\" (UID: \"e3c3d477-f4c2-4740-b15e-aea2e058197b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47842a96-ccbdc"
Oct 30 22:24:28.221744 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:28.221718    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3c3d477-f4c2-4740-b15e-aea2e058197b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47842a96-ccbdc\" (UID: \"e3c3d477-f4c2-4740-b15e-aea2e058197b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47842a96-ccbdc"
Oct 30 22:24:28.322943 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:28.322913    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3c3d477-f4c2-4740-b15e-aea2e058197b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47842a96-ccbdc\" (UID: \"e3c3d477-f4c2-4740-b15e-aea2e058197b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47842a96-ccbdc"
Oct 30 22:24:28.323126 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:28.323001    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3c3d477-f4c2-4740-b15e-aea2e058197b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47842a96-ccbdc\" (UID: \"e3c3d477-f4c2-4740-b15e-aea2e058197b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47842a96-ccbdc"
Oct 30 22:24:28.323126 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:28.323029    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bjkf\" (UniqueName: \"kubernetes.io/projected/e3c3d477-f4c2-4740-b15e-aea2e058197b-kube-api-access-2bjkf\") pod \"maas-api-revoke-sub-47842a96-ccbdc\" (UID: \"e3c3d477-f4c2-4740-b15e-aea2e058197b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47842a96-ccbdc"
Oct 30 22:24:28.323713 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:28.323687    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3c3d477-f4c2-4740-b15e-aea2e058197b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47842a96-ccbdc\" (UID: \"e3c3d477-f4c2-4740-b15e-aea2e058197b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47842a96-ccbdc"
Oct 30 22:24:28.325267 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:28.325245    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3c3d477-f4c2-4740-b15e-aea2e058197b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47842a96-ccbdc\" (UID: \"e3c3d477-f4c2-4740-b15e-aea2e058197b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47842a96-ccbdc"
Oct 30 22:24:28.377855 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:28.377819    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bjkf\" (UniqueName: \"kubernetes.io/projected/e3c3d477-f4c2-4740-b15e-aea2e058197b-kube-api-access-2bjkf\") pod \"maas-api-revoke-sub-47842a96-ccbdc\" (UID: \"e3c3d477-f4c2-4740-b15e-aea2e058197b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47842a96-ccbdc"
Oct 30 22:24:28.639024 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:28.638934    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47842a96-ccbdc"
Oct 30 22:24:28.903162 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:28.903075    5061 scope.go:122] "RemoveContainer" containerID="e44ff32b18531be8417e7c327ccefc08dc7776ab4ab4bc3efff775c162152b97"
Oct 30 22:24:28.915807 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:28.915769    5061 scope.go:122] "RemoveContainer" containerID="754b30b1837ce0326dde1b1761f7480add843ae9295e3531650e5fd333f8152e"
Oct 30 22:24:29.013633 ip-10-0-130-60 kubenswrapper[5061]: W1030 22:24:29.013597    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3c3d477_f4c2_4740_b15e_aea2e058197b.slice/crio-e69087f17388c43696f81eaad2faa32984369769170d4a913ce2c4b9d318ffdb WatchSource:0}: Error finding container e69087f17388c43696f81eaad2faa32984369769170d4a913ce2c4b9d318ffdb: Status 404 returned error can't find the container with id e69087f17388c43696f81eaad2faa32984369769170d4a913ce2c4b9d318ffdb
Oct 30 22:24:29.016518 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:29.015769    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47842a96-ccbdc"]
Oct 30 22:24:29.999971 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:29.999929    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47842a96-ccbdc" event={"ID":"e3c3d477-f4c2-4740-b15e-aea2e058197b","Type":"ContainerStarted","Data":"f6e138f9074b6788e957dff55a8390451591198a10f4fc7e67bbbeb7a820d27f"}
Oct 30 22:24:30.000494 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:29.999996    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47842a96-ccbdc" event={"ID":"e3c3d477-f4c2-4740-b15e-aea2e058197b","Type":"ContainerStarted","Data":"e69087f17388c43696f81eaad2faa32984369769170d4a913ce2c4b9d318ffdb"}
Oct 30 22:24:30.111266 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:30.111208    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47842a96-ccbdc" podStartSLOduration=3.111193458 podStartE2EDuration="3.111193458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:24:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:24:30.110511181 +0000 UTC m=+2523.723280942" watchObservedRunningTime="2026-10-30 22:24:30.111193458 +0000 UTC m=+2523.723963256"
Oct 30 22:24:31.011195 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:31.011164    5061 generic.go:356] "Generic (PLEG): container finished" podID="e3c3d477-f4c2-4740-b15e-aea2e058197b" containerID="f6e138f9074b6788e957dff55a8390451591198a10f4fc7e67bbbeb7a820d27f" exitCode=0
Oct 30 22:24:31.011586 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:31.011243    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47842a96-ccbdc" event={"ID":"e3c3d477-f4c2-4740-b15e-aea2e058197b","Type":"ContainerDied","Data":"f6e138f9074b6788e957dff55a8390451591198a10f4fc7e67bbbeb7a820d27f"}
Oct 30 22:24:32.166912 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:32.166883    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47842a96-ccbdc"
Oct 30 22:24:32.194645 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:32.194620    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3c3d477-f4c2-4740-b15e-aea2e058197b-kube-api-access-2bjkf\" (UniqueName: \"kubernetes.io/projected/e3c3d477-f4c2-4740-b15e-aea2e058197b-kube-api-access-2bjkf\") pod \"e3c3d477-f4c2-4740-b15e-aea2e058197b\" (UID: \"e3c3d477-f4c2-4740-b15e-aea2e058197b\") "
Oct 30 22:24:32.194772 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:32.194724    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e3c3d477-f4c2-4740-b15e-aea2e058197b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3c3d477-f4c2-4740-b15e-aea2e058197b-maas-api-xks-ca\") pod \"e3c3d477-f4c2-4740-b15e-aea2e058197b\" (UID: \"e3c3d477-f4c2-4740-b15e-aea2e058197b\") "
Oct 30 22:24:32.194772 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:32.194749    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3c3d477-f4c2-4740-b15e-aea2e058197b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3c3d477-f4c2-4740-b15e-aea2e058197b-maas-api-openshift-ca\") pod \"e3c3d477-f4c2-4740-b15e-aea2e058197b\" (UID: \"e3c3d477-f4c2-4740-b15e-aea2e058197b\") "
Oct 30 22:24:32.195159 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:32.195133    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3c3d477-f4c2-4740-b15e-aea2e058197b-maas-api-openshift-ca" pod "e3c3d477-f4c2-4740-b15e-aea2e058197b" (UID: "e3c3d477-f4c2-4740-b15e-aea2e058197b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:24:32.196723 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:32.196697    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3c3d477-f4c2-4740-b15e-aea2e058197b-kube-api-access-2bjkf" pod "e3c3d477-f4c2-4740-b15e-aea2e058197b" (UID: "e3c3d477-f4c2-4740-b15e-aea2e058197b"). InnerVolumeSpecName "kube-api-access-2bjkf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:24:32.196835 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:32.196797    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3c3d477-f4c2-4740-b15e-aea2e058197b-maas-api-xks-ca" pod "e3c3d477-f4c2-4740-b15e-aea2e058197b" (UID: "e3c3d477-f4c2-4740-b15e-aea2e058197b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:24:32.296234 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:32.296201    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bjkf\" (UniqueName: \"kubernetes.io/projected/e3c3d477-f4c2-4740-b15e-aea2e058197b-kube-api-access-2bjkf\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:24:32.296234 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:32.296230    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3c3d477-f4c2-4740-b15e-aea2e058197b-maas-api-xks-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:24:32.296234 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:32.296240    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3c3d477-f4c2-4740-b15e-aea2e058197b-maas-api-openshift-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:24:33.025611 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:33.025574    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47842a96-ccbdc"
Oct 30 22:24:33.025611 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:33.025597    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47842a96-ccbdc" event={"ID":"e3c3d477-f4c2-4740-b15e-aea2e058197b","Type":"ContainerDied","Data":"e69087f17388c43696f81eaad2faa32984369769170d4a913ce2c4b9d318ffdb"}
Oct 30 22:24:33.025846 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:33.025649    5061 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e69087f17388c43696f81eaad2faa32984369769170d4a913ce2c4b9d318ffdb"
Oct 30 22:24:39.359759 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:39.359724    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-17d809-vqztk"]
Oct 30 22:24:39.360570 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:39.360516    5061 cpu_manager.go:414] "containerMap: removing container" podUID="e3c3d477-f4c2-4740-b15e-aea2e058197b" containerName="revoke-keys"
Oct 30 22:24:39.360570 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:39.360546    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3c3d477-f4c2-4740-b15e-aea2e058197b" containerName="revoke-keys"
Oct 30 22:24:39.360754 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:39.360642    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3c3d477-f4c2-4740-b15e-aea2e058197b" containerName="revoke-keys"
Oct 30 22:24:39.364223 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:39.364206    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-17d809-vqztk"
Oct 30 22:24:39.413801 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:39.413772    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:24:39.460632 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:39.460600    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gdmfg\""
Oct 30 22:24:39.464675 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:39.464653    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-17d809-vqztk"]
Oct 30 22:24:39.477285 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:39.477259    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b79a5098-150c-48f1-a156-e54615c1be82-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-17d809-vqztk\" (UID: \"b79a5098-150c-48f1-a156-e54615c1be82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-17d809-vqztk"
Oct 30 22:24:39.477417 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:39.477400    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b79a5098-150c-48f1-a156-e54615c1be82-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-17d809-vqztk\" (UID: \"b79a5098-150c-48f1-a156-e54615c1be82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-17d809-vqztk"
Oct 30 22:24:39.477486 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:39.477451    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8brd4\" (UniqueName: \"kubernetes.io/projected/b79a5098-150c-48f1-a156-e54615c1be82-kube-api-access-8brd4\") pod \"maas-api-revoke-keys-e2e-worker-w4-17d809-vqztk\" (UID: \"b79a5098-150c-48f1-a156-e54615c1be82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-17d809-vqztk"
Oct 30 22:24:39.578080 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:39.578051    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b79a5098-150c-48f1-a156-e54615c1be82-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-17d809-vqztk\" (UID: \"b79a5098-150c-48f1-a156-e54615c1be82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-17d809-vqztk"
Oct 30 22:24:39.578229 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:39.578101    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8brd4\" (UniqueName: \"kubernetes.io/projected/b79a5098-150c-48f1-a156-e54615c1be82-kube-api-access-8brd4\") pod \"maas-api-revoke-keys-e2e-worker-w4-17d809-vqztk\" (UID: \"b79a5098-150c-48f1-a156-e54615c1be82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-17d809-vqztk"
Oct 30 22:24:39.578229 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:39.578171    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b79a5098-150c-48f1-a156-e54615c1be82-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-17d809-vqztk\" (UID: \"b79a5098-150c-48f1-a156-e54615c1be82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-17d809-vqztk"
Oct 30 22:24:39.578800 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:39.578775    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b79a5098-150c-48f1-a156-e54615c1be82-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-17d809-vqztk\" (UID: \"b79a5098-150c-48f1-a156-e54615c1be82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-17d809-vqztk"
Oct 30 22:24:39.580488 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:39.580466    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b79a5098-150c-48f1-a156-e54615c1be82-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-17d809-vqztk\" (UID: \"b79a5098-150c-48f1-a156-e54615c1be82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-17d809-vqztk"
Oct 30 22:24:39.632646 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:39.632589    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8brd4\" (UniqueName: \"kubernetes.io/projected/b79a5098-150c-48f1-a156-e54615c1be82-kube-api-access-8brd4\") pod \"maas-api-revoke-keys-e2e-worker-w4-17d809-vqztk\" (UID: \"b79a5098-150c-48f1-a156-e54615c1be82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-17d809-vqztk"
Oct 30 22:24:39.677965 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:39.677938    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-17d809-vqztk"
Oct 30 22:24:40.055335 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:40.055306    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-17d809-vqztk"]
Oct 30 22:24:40.055913 ip-10-0-130-60 kubenswrapper[5061]: W1030 22:24:40.055884    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb79a5098_150c_48f1_a156_e54615c1be82.slice/crio-2ff4eae7f7e815607bc8033ddbfc25132db3e586f66710f70a3b706aa561c6e2 WatchSource:0}: Error finding container 2ff4eae7f7e815607bc8033ddbfc25132db3e586f66710f70a3b706aa561c6e2: Status 404 returned error can't find the container with id 2ff4eae7f7e815607bc8033ddbfc25132db3e586f66710f70a3b706aa561c6e2
Oct 30 22:24:40.074602 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:40.074574    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-17d809-vqztk" event={"ID":"b79a5098-150c-48f1-a156-e54615c1be82","Type":"ContainerStarted","Data":"2ff4eae7f7e815607bc8033ddbfc25132db3e586f66710f70a3b706aa561c6e2"}
Oct 30 22:24:41.085864 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:41.085829    5061 generic.go:356] "Generic (PLEG): container finished" podID="b79a5098-150c-48f1-a156-e54615c1be82" containerID="616fdec68611feded71059e1604adb7d684a5aec6071420cfeeaa26ee5d14046" exitCode=0
Oct 30 22:24:41.086258 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:41.085911    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-17d809-vqztk" event={"ID":"b79a5098-150c-48f1-a156-e54615c1be82","Type":"ContainerDied","Data":"616fdec68611feded71059e1604adb7d684a5aec6071420cfeeaa26ee5d14046"}
Oct 30 22:24:42.253067 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:42.253043    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-17d809-vqztk"
Oct 30 22:24:42.309634 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:42.309603    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b79a5098-150c-48f1-a156-e54615c1be82-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b79a5098-150c-48f1-a156-e54615c1be82-maas-api-openshift-ca\") pod \"b79a5098-150c-48f1-a156-e54615c1be82\" (UID: \"b79a5098-150c-48f1-a156-e54615c1be82\") "
Oct 30 22:24:42.309791 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:42.309669    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b79a5098-150c-48f1-a156-e54615c1be82-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b79a5098-150c-48f1-a156-e54615c1be82-maas-api-xks-ca\") pod \"b79a5098-150c-48f1-a156-e54615c1be82\" (UID: \"b79a5098-150c-48f1-a156-e54615c1be82\") "
Oct 30 22:24:42.309791 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:42.309725    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b79a5098-150c-48f1-a156-e54615c1be82-kube-api-access-8brd4\" (UniqueName: \"kubernetes.io/projected/b79a5098-150c-48f1-a156-e54615c1be82-kube-api-access-8brd4\") pod \"b79a5098-150c-48f1-a156-e54615c1be82\" (UID: \"b79a5098-150c-48f1-a156-e54615c1be82\") "
Oct 30 22:24:42.309974 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:42.309948    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b79a5098-150c-48f1-a156-e54615c1be82-maas-api-openshift-ca" pod "b79a5098-150c-48f1-a156-e54615c1be82" (UID: "b79a5098-150c-48f1-a156-e54615c1be82"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:24:42.310172 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:42.310154    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b79a5098-150c-48f1-a156-e54615c1be82-maas-api-openshift-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:24:42.311855 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:42.311833    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b79a5098-150c-48f1-a156-e54615c1be82-maas-api-xks-ca" pod "b79a5098-150c-48f1-a156-e54615c1be82" (UID: "b79a5098-150c-48f1-a156-e54615c1be82"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:24:42.311960 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:42.311861    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b79a5098-150c-48f1-a156-e54615c1be82-kube-api-access-8brd4" pod "b79a5098-150c-48f1-a156-e54615c1be82" (UID: "b79a5098-150c-48f1-a156-e54615c1be82"). InnerVolumeSpecName "kube-api-access-8brd4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:24:42.411143 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:42.411060    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b79a5098-150c-48f1-a156-e54615c1be82-maas-api-xks-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:24:42.411143 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:42.411093    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8brd4\" (UniqueName: \"kubernetes.io/projected/b79a5098-150c-48f1-a156-e54615c1be82-kube-api-access-8brd4\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:24:43.102433 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:43.102397    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-17d809-vqztk"
Oct 30 22:24:43.102615 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:43.102396    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-17d809-vqztk" event={"ID":"b79a5098-150c-48f1-a156-e54615c1be82","Type":"ContainerDied","Data":"2ff4eae7f7e815607bc8033ddbfc25132db3e586f66710f70a3b706aa561c6e2"}
Oct 30 22:24:43.102615 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:43.102519    5061 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2ff4eae7f7e815607bc8033ddbfc25132db3e586f66710f70a3b706aa561c6e2"
Oct 30 22:24:43.759148 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:43.759116    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-17d809/distinct-w4-17d809-kserve-6b8f5c48cd-lhrwp"]
Oct 30 22:24:43.806220 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:43.806190    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-17d809/premium-w4-17d809-kserve-7c9bdbbcbc-5mtmm"]
Oct 30 22:24:43.809871 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:43.809847    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-17d809/simulator-w4-17d809-kserve-78d67b6f4-7hsvb"]
Oct 30 22:24:45.547959 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:45.547915    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-5be0d1-jzdjw"]
Oct 30 22:24:45.595049 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:45.595011    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-5be0d1-jzdjw"]
Oct 30 22:24:46.529489 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:46.529453    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17eb78ae-158c-49f2-8ff2-b7f1a7918a82" path="/var/lib/kubelet/pods/17eb78ae-158c-49f2-8ff2-b7f1a7918a82/volumes"
Oct 30 22:24:49.052886 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:49.052847    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dae567fc-z9w97"]
Oct 30 22:24:49.099153 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:49.099117    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dae567fc-z9w97"]
Oct 30 22:24:49.102457 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:49.102427    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91211992-5vwmg"]
Oct 30 22:24:49.105441 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:49.105417    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91211992-5vwmg"]
Oct 30 22:24:50.530024 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:50.529989    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40ac0057-5b20-4af0-89ea-037f7400146a" path="/var/lib/kubelet/pods/40ac0057-5b20-4af0-89ea-037f7400146a/volumes"
Oct 30 22:24:50.530723 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:50.530698    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f45e5e6c-e786-4b83-ab69-a25594e59764" path="/var/lib/kubelet/pods/f45e5e6c-e786-4b83-ab69-a25594e59764/volumes"
Oct 30 22:24:50.821468 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:50.821434    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-17d809-686d4b9688-4zrv4"]
Oct 30 22:24:53.462670 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:53.462632    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rsj7x"]
Oct 30 22:24:53.463176 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:53.462897    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rsj7x" podUID="fa405318-3b68-43f6-bfe4-d1fc80834e11" containerName="istio-proxy" containerID="cri-o://2baa7f9fab031ecca26095d5131691cc319d4096201108d71660d90906783439" gracePeriod=30
Oct 30 22:24:54.667435 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:24:54.667395    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 22:24:54.681803 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:54.681775    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8449ec64-vv9hj"]
Oct 30 22:24:54.682455 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:54.682435    5061 cpu_manager.go:414] "containerMap: removing container" podUID="b79a5098-150c-48f1-a156-e54615c1be82" containerName="revoke-keys"
Oct 30 22:24:54.682568 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:54.682457    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b79a5098-150c-48f1-a156-e54615c1be82" containerName="revoke-keys"
Oct 30 22:24:54.682651 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:54.682637    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="b79a5098-150c-48f1-a156-e54615c1be82" containerName="revoke-keys"
Oct 30 22:24:54.686328 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:54.686309    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8449ec64-vv9hj"
Oct 30 22:24:54.737259 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:54.737230    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75126b1f-hzlhv"]
Oct 30 22:24:54.741521 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:54.741501    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75126b1f-hzlhv"
Oct 30 22:24:54.780559 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:54.780507    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:24:54.780559 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:54.780512    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gdmfg\""
Oct 30 22:24:54.785968 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:54.785943    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8449ec64-vv9hj"]
Oct 30 22:24:54.798756 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:54.798732    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86f24e81-ee68-4d6f-b1f7-c55ff25c872d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8449ec64-vv9hj\" (UID: \"86f24e81-ee68-4d6f-b1f7-c55ff25c872d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8449ec64-vv9hj"
Oct 30 22:24:54.798883 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:54.798808    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kt6sz\" (UniqueName: \"kubernetes.io/projected/86f24e81-ee68-4d6f-b1f7-c55ff25c872d-kube-api-access-kt6sz\") pod \"maas-api-revoke-sub-8449ec64-vv9hj\" (UID: \"86f24e81-ee68-4d6f-b1f7-c55ff25c872d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8449ec64-vv9hj"
Oct 30 22:24:54.798929 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:54.798922    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86f24e81-ee68-4d6f-b1f7-c55ff25c872d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8449ec64-vv9hj\" (UID: \"86f24e81-ee68-4d6f-b1f7-c55ff25c872d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8449ec64-vv9hj"
Oct 30 22:24:54.834382 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:54.834348    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75126b1f-hzlhv"]
Oct 30 22:24:54.900144 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:54.900101    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17180b54-43c4-4af7-af97-ae8e37a2ba46-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75126b1f-hzlhv\" (UID: \"17180b54-43c4-4af7-af97-ae8e37a2ba46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75126b1f-hzlhv"
Oct 30 22:24:54.900302 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:54.900195    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86f24e81-ee68-4d6f-b1f7-c55ff25c872d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8449ec64-vv9hj\" (UID: \"86f24e81-ee68-4d6f-b1f7-c55ff25c872d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8449ec64-vv9hj"
Oct 30 22:24:54.900302 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:54.900266    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6kcb\" (UniqueName: \"kubernetes.io/projected/17180b54-43c4-4af7-af97-ae8e37a2ba46-kube-api-access-b6kcb\") pod \"maas-api-revoke-sub-75126b1f-hzlhv\" (UID: \"17180b54-43c4-4af7-af97-ae8e37a2ba46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75126b1f-hzlhv"
Oct 30 22:24:54.900428 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:54.900353    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86f24e81-ee68-4d6f-b1f7-c55ff25c872d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8449ec64-vv9hj\" (UID: \"86f24e81-ee68-4d6f-b1f7-c55ff25c872d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8449ec64-vv9hj"
Oct 30 22:24:54.900473 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:54.900430    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17180b54-43c4-4af7-af97-ae8e37a2ba46-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75126b1f-hzlhv\" (UID: \"17180b54-43c4-4af7-af97-ae8e37a2ba46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75126b1f-hzlhv"
Oct 30 22:24:54.900519 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:54.900502    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kt6sz\" (UniqueName: \"kubernetes.io/projected/86f24e81-ee68-4d6f-b1f7-c55ff25c872d-kube-api-access-kt6sz\") pod \"maas-api-revoke-sub-8449ec64-vv9hj\" (UID: \"86f24e81-ee68-4d6f-b1f7-c55ff25c872d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8449ec64-vv9hj"
Oct 30 22:24:54.901048 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:54.901022    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86f24e81-ee68-4d6f-b1f7-c55ff25c872d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8449ec64-vv9hj\" (UID: \"86f24e81-ee68-4d6f-b1f7-c55ff25c872d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8449ec64-vv9hj"
Oct 30 22:24:54.902663 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:54.902643    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86f24e81-ee68-4d6f-b1f7-c55ff25c872d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8449ec64-vv9hj\" (UID: \"86f24e81-ee68-4d6f-b1f7-c55ff25c872d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8449ec64-vv9hj"
Oct 30 22:24:54.956030 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:54.955963    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kt6sz\" (UniqueName: \"kubernetes.io/projected/86f24e81-ee68-4d6f-b1f7-c55ff25c872d-kube-api-access-kt6sz\") pod \"maas-api-revoke-sub-8449ec64-vv9hj\" (UID: \"86f24e81-ee68-4d6f-b1f7-c55ff25c872d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8449ec64-vv9hj"
Oct 30 22:24:55.001302 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:55.001272    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8449ec64-vv9hj"
Oct 30 22:24:55.001681 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:55.001658    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6kcb\" (UniqueName: \"kubernetes.io/projected/17180b54-43c4-4af7-af97-ae8e37a2ba46-kube-api-access-b6kcb\") pod \"maas-api-revoke-sub-75126b1f-hzlhv\" (UID: \"17180b54-43c4-4af7-af97-ae8e37a2ba46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75126b1f-hzlhv"
Oct 30 22:24:55.001764 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:55.001725    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17180b54-43c4-4af7-af97-ae8e37a2ba46-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75126b1f-hzlhv\" (UID: \"17180b54-43c4-4af7-af97-ae8e37a2ba46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75126b1f-hzlhv"
Oct 30 22:24:55.001929 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:55.001911    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17180b54-43c4-4af7-af97-ae8e37a2ba46-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75126b1f-hzlhv\" (UID: \"17180b54-43c4-4af7-af97-ae8e37a2ba46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75126b1f-hzlhv"
Oct 30 22:24:55.002505 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:55.002480    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17180b54-43c4-4af7-af97-ae8e37a2ba46-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75126b1f-hzlhv\" (UID: \"17180b54-43c4-4af7-af97-ae8e37a2ba46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75126b1f-hzlhv"
Oct 30 22:24:55.004566 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:55.004433    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17180b54-43c4-4af7-af97-ae8e37a2ba46-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75126b1f-hzlhv\" (UID: \"17180b54-43c4-4af7-af97-ae8e37a2ba46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75126b1f-hzlhv"
Oct 30 22:24:55.058228 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:55.058200    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6kcb\" (UniqueName: \"kubernetes.io/projected/17180b54-43c4-4af7-af97-ae8e37a2ba46-kube-api-access-b6kcb\") pod \"maas-api-revoke-sub-75126b1f-hzlhv\" (UID: \"17180b54-43c4-4af7-af97-ae8e37a2ba46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75126b1f-hzlhv"
Oct 30 22:24:55.357076 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:55.357043    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75126b1f-hzlhv"
Oct 30 22:24:55.382106 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:55.382083    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8449ec64-vv9hj"]
Oct 30 22:24:55.382758 ip-10-0-130-60 kubenswrapper[5061]: W1030 22:24:55.382729    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86f24e81_ee68_4d6f_b1f7_c55ff25c872d.slice/crio-93a41795455e1b31f1ac447fad927d806a129d586a8d20ee158d59795813b152 WatchSource:0}: Error finding container 93a41795455e1b31f1ac447fad927d806a129d586a8d20ee158d59795813b152: Status 404 returned error can't find the container with id 93a41795455e1b31f1ac447fad927d806a129d586a8d20ee158d59795813b152
Oct 30 22:24:55.734167 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:55.734135    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75126b1f-hzlhv"]
Oct 30 22:24:55.734507 ip-10-0-130-60 kubenswrapper[5061]: W1030 22:24:55.734146    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17180b54_43c4_4af7_af97_ae8e37a2ba46.slice/crio-65edd48ca2b69b6cdaa4b9100213646b0020e36773f71e8797d4c0a6bc2daf47 WatchSource:0}: Error finding container 65edd48ca2b69b6cdaa4b9100213646b0020e36773f71e8797d4c0a6bc2daf47: Status 404 returned error can't find the container with id 65edd48ca2b69b6cdaa4b9100213646b0020e36773f71e8797d4c0a6bc2daf47
Oct 30 22:24:55.821760 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:55.821720    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-17d809-686d4b9688-4zrv4" podUID="71fc13bb-d3e2-4a61-9942-4195b36d227c" containerName="payload-processing" containerID="cri-o://80aa3195220ba7a419be23e77dac98eebdc4d67dde595d19e0af8354740743b9" gracePeriod=25
Oct 30 22:24:56.200147 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:56.200114    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75126b1f-hzlhv" event={"ID":"17180b54-43c4-4af7-af97-ae8e37a2ba46","Type":"ContainerStarted","Data":"7b1113453a932d05bbd87be2a92526a1a2105119e09762f7c5c92398c2b4918a"}
Oct 30 22:24:56.200337 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:56.200171    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75126b1f-hzlhv" event={"ID":"17180b54-43c4-4af7-af97-ae8e37a2ba46","Type":"ContainerStarted","Data":"65edd48ca2b69b6cdaa4b9100213646b0020e36773f71e8797d4c0a6bc2daf47"}
Oct 30 22:24:56.201871 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:56.201841    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8449ec64-vv9hj" event={"ID":"86f24e81-ee68-4d6f-b1f7-c55ff25c872d","Type":"ContainerStarted","Data":"60800004c41122a730f6d9afab0b76e0bb7318d24aa7b8f0f61646154eb89425"}
Oct 30 22:24:56.201991 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:56.201894    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8449ec64-vv9hj" event={"ID":"86f24e81-ee68-4d6f-b1f7-c55ff25c872d","Type":"ContainerStarted","Data":"93a41795455e1b31f1ac447fad927d806a129d586a8d20ee158d59795813b152"}
Oct 30 22:24:56.312112 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:56.312060    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75126b1f-hzlhv" podStartSLOduration=2.312048089 podStartE2EDuration="2.312048089s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:24:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:24:56.30852006 +0000 UTC m=+2549.921289837" watchObservedRunningTime="2026-10-30 22:24:56.312048089 +0000 UTC m=+2549.924817847"
Oct 30 22:24:56.422377 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:56.422272    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8449ec64-vv9hj" podStartSLOduration=2.42225835 podStartE2EDuration="2.42225835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:24:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:24:56.419538232 +0000 UTC m=+2550.032307993" watchObservedRunningTime="2026-10-30 22:24:56.42225835 +0000 UTC m=+2550.035028108"
Oct 30 22:24:57.210809 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:57.210775    5061 generic.go:356] "Generic (PLEG): container finished" podID="17180b54-43c4-4af7-af97-ae8e37a2ba46" containerID="7b1113453a932d05bbd87be2a92526a1a2105119e09762f7c5c92398c2b4918a" exitCode=0
Oct 30 22:24:57.211250 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:57.210861    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75126b1f-hzlhv" event={"ID":"17180b54-43c4-4af7-af97-ae8e37a2ba46","Type":"ContainerDied","Data":"7b1113453a932d05bbd87be2a92526a1a2105119e09762f7c5c92398c2b4918a"}
Oct 30 22:24:57.212471 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:57.212446    5061 generic.go:356] "Generic (PLEG): container finished" podID="86f24e81-ee68-4d6f-b1f7-c55ff25c872d" containerID="60800004c41122a730f6d9afab0b76e0bb7318d24aa7b8f0f61646154eb89425" exitCode=0
Oct 30 22:24:57.212641 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:57.212557    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8449ec64-vv9hj" event={"ID":"86f24e81-ee68-4d6f-b1f7-c55ff25c872d","Type":"ContainerDied","Data":"60800004c41122a730f6d9afab0b76e0bb7318d24aa7b8f0f61646154eb89425"}
Oct 30 22:24:58.421087 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:58.421059    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75126b1f-hzlhv"
Oct 30 22:24:58.424600 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:58.424575    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8449ec64-vv9hj"
Oct 30 22:24:58.443162 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:58.443133    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86f24e81-ee68-4d6f-b1f7-c55ff25c872d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86f24e81-ee68-4d6f-b1f7-c55ff25c872d-maas-api-xks-ca\") pod \"86f24e81-ee68-4d6f-b1f7-c55ff25c872d\" (UID: \"86f24e81-ee68-4d6f-b1f7-c55ff25c872d\") "
Oct 30 22:24:58.443392 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:58.443238    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86f24e81-ee68-4d6f-b1f7-c55ff25c872d-kube-api-access-kt6sz\" (UniqueName: \"kubernetes.io/projected/86f24e81-ee68-4d6f-b1f7-c55ff25c872d-kube-api-access-kt6sz\") pod \"86f24e81-ee68-4d6f-b1f7-c55ff25c872d\" (UID: \"86f24e81-ee68-4d6f-b1f7-c55ff25c872d\") "
Oct 30 22:24:58.443392 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:58.443282    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17180b54-43c4-4af7-af97-ae8e37a2ba46-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17180b54-43c4-4af7-af97-ae8e37a2ba46-maas-api-openshift-ca\") pod \"17180b54-43c4-4af7-af97-ae8e37a2ba46\" (UID: \"17180b54-43c4-4af7-af97-ae8e37a2ba46\") "
Oct 30 22:24:58.443392 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:58.443332    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17180b54-43c4-4af7-af97-ae8e37a2ba46-kube-api-access-b6kcb\" (UniqueName: \"kubernetes.io/projected/17180b54-43c4-4af7-af97-ae8e37a2ba46-kube-api-access-b6kcb\") pod \"17180b54-43c4-4af7-af97-ae8e37a2ba46\" (UID: \"17180b54-43c4-4af7-af97-ae8e37a2ba46\") "
Oct 30 22:24:58.443643 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:58.443413    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17180b54-43c4-4af7-af97-ae8e37a2ba46-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17180b54-43c4-4af7-af97-ae8e37a2ba46-maas-api-xks-ca\") pod \"17180b54-43c4-4af7-af97-ae8e37a2ba46\" (UID: \"17180b54-43c4-4af7-af97-ae8e37a2ba46\") "
Oct 30 22:24:58.443643 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:58.443458    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86f24e81-ee68-4d6f-b1f7-c55ff25c872d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86f24e81-ee68-4d6f-b1f7-c55ff25c872d-maas-api-openshift-ca\") pod \"86f24e81-ee68-4d6f-b1f7-c55ff25c872d\" (UID: \"86f24e81-ee68-4d6f-b1f7-c55ff25c872d\") "
Oct 30 22:24:58.444051 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:58.443765    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17180b54-43c4-4af7-af97-ae8e37a2ba46-maas-api-openshift-ca" pod "17180b54-43c4-4af7-af97-ae8e37a2ba46" (UID: "17180b54-43c4-4af7-af97-ae8e37a2ba46"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:24:58.444340 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:58.444138    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17180b54-43c4-4af7-af97-ae8e37a2ba46-maas-api-openshift-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:24:58.444495 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:58.444471    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86f24e81-ee68-4d6f-b1f7-c55ff25c872d-maas-api-openshift-ca" pod "86f24e81-ee68-4d6f-b1f7-c55ff25c872d" (UID: "86f24e81-ee68-4d6f-b1f7-c55ff25c872d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:24:58.445883 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:58.445847    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86f24e81-ee68-4d6f-b1f7-c55ff25c872d-maas-api-xks-ca" pod "86f24e81-ee68-4d6f-b1f7-c55ff25c872d" (UID: "86f24e81-ee68-4d6f-b1f7-c55ff25c872d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:24:58.446069 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:58.446048    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17180b54-43c4-4af7-af97-ae8e37a2ba46-kube-api-access-b6kcb" pod "17180b54-43c4-4af7-af97-ae8e37a2ba46" (UID: "17180b54-43c4-4af7-af97-ae8e37a2ba46"). InnerVolumeSpecName "kube-api-access-b6kcb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:24:58.446825 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:58.446802    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86f24e81-ee68-4d6f-b1f7-c55ff25c872d-kube-api-access-kt6sz" pod "86f24e81-ee68-4d6f-b1f7-c55ff25c872d" (UID: "86f24e81-ee68-4d6f-b1f7-c55ff25c872d"). InnerVolumeSpecName "kube-api-access-kt6sz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:24:58.448006 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:58.447987    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17180b54-43c4-4af7-af97-ae8e37a2ba46-maas-api-xks-ca" pod "17180b54-43c4-4af7-af97-ae8e37a2ba46" (UID: "17180b54-43c4-4af7-af97-ae8e37a2ba46"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:24:58.545782 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:58.545753    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kt6sz\" (UniqueName: \"kubernetes.io/projected/86f24e81-ee68-4d6f-b1f7-c55ff25c872d-kube-api-access-kt6sz\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:24:58.545901 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:58.545878    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6kcb\" (UniqueName: \"kubernetes.io/projected/17180b54-43c4-4af7-af97-ae8e37a2ba46-kube-api-access-b6kcb\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:24:58.545951 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:58.545903    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17180b54-43c4-4af7-af97-ae8e37a2ba46-maas-api-xks-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:24:58.545951 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:58.545919    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86f24e81-ee68-4d6f-b1f7-c55ff25c872d-maas-api-openshift-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:24:58.545951 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:58.545935    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86f24e81-ee68-4d6f-b1f7-c55ff25c872d-maas-api-xks-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:24:58.725562 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:58.725519    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rsj7x"
Oct 30 22:24:58.748628 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:58.748578    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa405318-3b68-43f6-bfe4-d1fc80834e11-kube-api-access-srg75\" (UniqueName: \"kubernetes.io/projected/fa405318-3b68-43f6-bfe4-d1fc80834e11-kube-api-access-srg75\") pod \"fa405318-3b68-43f6-bfe4-d1fc80834e11\" (UID: \"fa405318-3b68-43f6-bfe4-d1fc80834e11\") "
Oct 30 22:24:58.748628 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:58.748625    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fa405318-3b68-43f6-bfe4-d1fc80834e11-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa405318-3b68-43f6-bfe4-d1fc80834e11-istio-podinfo\") pod \"fa405318-3b68-43f6-bfe4-d1fc80834e11\" (UID: \"fa405318-3b68-43f6-bfe4-d1fc80834e11\") "
Oct 30 22:24:58.748753 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:58.748652    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa405318-3b68-43f6-bfe4-d1fc80834e11-istio-token\" (UniqueName: \"kubernetes.io/projected/fa405318-3b68-43f6-bfe4-d1fc80834e11-istio-token\") pod \"fa405318-3b68-43f6-bfe4-d1fc80834e11\" (UID: \"fa405318-3b68-43f6-bfe4-d1fc80834e11\") "
Oct 30 22:24:58.748753 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:58.748685    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa405318-3b68-43f6-bfe4-d1fc80834e11-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa405318-3b68-43f6-bfe4-d1fc80834e11-credential-socket\") pod \"fa405318-3b68-43f6-bfe4-d1fc80834e11\" (UID: \"fa405318-3b68-43f6-bfe4-d1fc80834e11\") "
Oct 30 22:24:58.748753 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:58.748703    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa405318-3b68-43f6-bfe4-d1fc80834e11-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa405318-3b68-43f6-bfe4-d1fc80834e11-workload-socket\") pod \"fa405318-3b68-43f6-bfe4-d1fc80834e11\" (UID: \"fa405318-3b68-43f6-bfe4-d1fc80834e11\") "
Oct 30 22:24:58.748910 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:58.748765    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa405318-3b68-43f6-bfe4-d1fc80834e11-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa405318-3b68-43f6-bfe4-d1fc80834e11-workload-certs\") pod \"fa405318-3b68-43f6-bfe4-d1fc80834e11\" (UID: \"fa405318-3b68-43f6-bfe4-d1fc80834e11\") "
Oct 30 22:24:58.748910 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:58.748827    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa405318-3b68-43f6-bfe4-d1fc80834e11-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa405318-3b68-43f6-bfe4-d1fc80834e11-istio-data\") pod \"fa405318-3b68-43f6-bfe4-d1fc80834e11\" (UID: \"fa405318-3b68-43f6-bfe4-d1fc80834e11\") "
Oct 30 22:24:58.748910 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:58.748895    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa405318-3b68-43f6-bfe4-d1fc80834e11-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa405318-3b68-43f6-bfe4-d1fc80834e11-istiod-ca-cert\") pod \"fa405318-3b68-43f6-bfe4-d1fc80834e11\" (UID: \"fa405318-3b68-43f6-bfe4-d1fc80834e11\") "
Oct 30 22:24:58.749087 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:58.748941    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa405318-3b68-43f6-bfe4-d1fc80834e11-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa405318-3b68-43f6-bfe4-d1fc80834e11-istio-envoy\") pod \"fa405318-3b68-43f6-bfe4-d1fc80834e11\" (UID: \"fa405318-3b68-43f6-bfe4-d1fc80834e11\") "
Oct 30 22:24:58.749087 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:58.748968    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa405318-3b68-43f6-bfe4-d1fc80834e11-credential-socket" pod "fa405318-3b68-43f6-bfe4-d1fc80834e11" (UID: "fa405318-3b68-43f6-bfe4-d1fc80834e11"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:24:58.749225 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:58.749165    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa405318-3b68-43f6-bfe4-d1fc80834e11-istio-data" pod "fa405318-3b68-43f6-bfe4-d1fc80834e11" (UID: "fa405318-3b68-43f6-bfe4-d1fc80834e11"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:24:58.749366 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:58.749284    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa405318-3b68-43f6-bfe4-d1fc80834e11-workload-certs" pod "fa405318-3b68-43f6-bfe4-d1fc80834e11" (UID: "fa405318-3b68-43f6-bfe4-d1fc80834e11"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:24:58.749366 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:58.749328    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa405318-3b68-43f6-bfe4-d1fc80834e11-istiod-ca-cert" pod "fa405318-3b68-43f6-bfe4-d1fc80834e11" (UID: "fa405318-3b68-43f6-bfe4-d1fc80834e11"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:24:58.749366 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:58.749336    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa405318-3b68-43f6-bfe4-d1fc80834e11-workload-socket" pod "fa405318-3b68-43f6-bfe4-d1fc80834e11" (UID: "fa405318-3b68-43f6-bfe4-d1fc80834e11"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:24:58.749649 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:58.749623    5061 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa405318-3b68-43f6-bfe4-d1fc80834e11-istiod-ca-cert\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:24:58.749781 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:58.749651    5061 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa405318-3b68-43f6-bfe4-d1fc80834e11-credential-socket\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:24:58.749781 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:58.749667    5061 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa405318-3b68-43f6-bfe4-d1fc80834e11-workload-socket\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:24:58.749781 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:58.749682    5061 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa405318-3b68-43f6-bfe4-d1fc80834e11-workload-certs\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:24:58.749781 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:58.749696    5061 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa405318-3b68-43f6-bfe4-d1fc80834e11-istio-data\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:24:58.750896 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:58.750872    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa405318-3b68-43f6-bfe4-d1fc80834e11-kube-api-access-srg75" pod "fa405318-3b68-43f6-bfe4-d1fc80834e11" (UID: "fa405318-3b68-43f6-bfe4-d1fc80834e11"). InnerVolumeSpecName "kube-api-access-srg75". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:24:58.751008 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:58.750988    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fa405318-3b68-43f6-bfe4-d1fc80834e11-istio-podinfo" pod "fa405318-3b68-43f6-bfe4-d1fc80834e11" (UID: "fa405318-3b68-43f6-bfe4-d1fc80834e11"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:24:58.751071 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:58.751006    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa405318-3b68-43f6-bfe4-d1fc80834e11-istio-token" pod "fa405318-3b68-43f6-bfe4-d1fc80834e11" (UID: "fa405318-3b68-43f6-bfe4-d1fc80834e11"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:24:58.751170 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:58.751153    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa405318-3b68-43f6-bfe4-d1fc80834e11-istio-envoy" pod "fa405318-3b68-43f6-bfe4-d1fc80834e11" (UID: "fa405318-3b68-43f6-bfe4-d1fc80834e11"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:24:58.780596 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:58.780566    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-17d809/distinct-w4-17d809-kserve-6b8f5c48cd-lhrwp" podUID="ef5e8691-42e1-471b-ba7d-90881028eca7" containerName="main" containerID="cri-o://a95e9b0f3ae527aeb8d1e8c8f557fc55453a836cd2574420c5a0e220bebd705a" gracePeriod=45
Oct 30 22:24:58.826994 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:58.826958    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-17d809/premium-w4-17d809-kserve-7c9bdbbcbc-5mtmm" podUID="1e0eca81-7db1-482b-8dd6-5b28fb89e010" containerName="main" containerID="cri-o://81f3a64544cb556ee2539afd8521fc7525f84b418f74c42af6074580ef77e5f7" gracePeriod=45
Oct 30 22:24:58.830243 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:58.830215    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-17d809/simulator-w4-17d809-kserve-78d67b6f4-7hsvb" podUID="e7337cf8-4f59-4afb-9b50-6b6c9fa4a773" containerName="main" containerID="cri-o://92fd7af56c5fb28daced7d6a9e574f44e4eed09cf90c8e4d8851bc2a78c11333" gracePeriod=45
Oct 30 22:24:58.850891 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:58.850867    5061 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa405318-3b68-43f6-bfe4-d1fc80834e11-istio-envoy\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:24:58.850891 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:58.850893    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srg75\" (UniqueName: \"kubernetes.io/projected/fa405318-3b68-43f6-bfe4-d1fc80834e11-kube-api-access-srg75\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:24:58.851017 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:58.850902    5061 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa405318-3b68-43f6-bfe4-d1fc80834e11-istio-podinfo\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:24:58.851017 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:58.850911    5061 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa405318-3b68-43f6-bfe4-d1fc80834e11-istio-token\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:24:58.942056 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:58.942018    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-17d809/simulator-w4-17d809-kserve-78d67b6f4-7hsvb" podUID="e7337cf8-4f59-4afb-9b50-6b6c9fa4a773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": dial tcp 10.132.0.68:8000: connect: connection refused"
Oct 30 22:24:59.098631 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:59.098596    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-17d809/simulator-w4-17d809-kserve-78d67b6f4-7hsvb"
Oct 30 22:24:59.152104 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:59.152078    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-17d809/premium-w4-17d809-kserve-7c9bdbbcbc-5mtmm"
Oct 30 22:24:59.153750 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:59.153719    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7337cf8-4f59-4afb-9b50-6b6c9fa4a773-kube-api-access-9ngwt\" (UniqueName: \"kubernetes.io/projected/e7337cf8-4f59-4afb-9b50-6b6c9fa4a773-kube-api-access-9ngwt\") pod \"e7337cf8-4f59-4afb-9b50-6b6c9fa4a773\" (UID: \"e7337cf8-4f59-4afb-9b50-6b6c9fa4a773\") "
Oct 30 22:24:59.153914 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:59.153829    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7337cf8-4f59-4afb-9b50-6b6c9fa4a773-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7337cf8-4f59-4afb-9b50-6b6c9fa4a773-dshm\") pod \"e7337cf8-4f59-4afb-9b50-6b6c9fa4a773\" (UID: \"e7337cf8-4f59-4afb-9b50-6b6c9fa4a773\") "
Oct 30 22:24:59.153994 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:59.153914    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7337cf8-4f59-4afb-9b50-6b6c9fa4a773-tls-certs\" (UniqueName: \"kubernetes.io/secret/e7337cf8-4f59-4afb-9b50-6b6c9fa4a773-tls-certs\") pod \"e7337cf8-4f59-4afb-9b50-6b6c9fa4a773\" (UID: \"e7337cf8-4f59-4afb-9b50-6b6c9fa4a773\") "
Oct 30 22:24:59.153994 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:59.153953    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7337cf8-4f59-4afb-9b50-6b6c9fa4a773-home\" (UniqueName: \"kubernetes.io/empty-dir/e7337cf8-4f59-4afb-9b50-6b6c9fa4a773-home\") pod \"e7337cf8-4f59-4afb-9b50-6b6c9fa4a773\" (UID: \"e7337cf8-4f59-4afb-9b50-6b6c9fa4a773\") "
Oct 30 22:24:59.154118 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:59.154010    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7337cf8-4f59-4afb-9b50-6b6c9fa4a773-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7337cf8-4f59-4afb-9b50-6b6c9fa4a773-tmp-dir\") pod \"e7337cf8-4f59-4afb-9b50-6b6c9fa4a773\" (UID: \"e7337cf8-4f59-4afb-9b50-6b6c9fa4a773\") "
Oct 30 22:24:59.154118 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:59.154081    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7337cf8-4f59-4afb-9b50-6b6c9fa4a773-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7337cf8-4f59-4afb-9b50-6b6c9fa4a773-model-cache\") pod \"e7337cf8-4f59-4afb-9b50-6b6c9fa4a773\" (UID: \"e7337cf8-4f59-4afb-9b50-6b6c9fa4a773\") "
Oct 30 22:24:59.155583 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:59.155288    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7337cf8-4f59-4afb-9b50-6b6c9fa4a773-tmp-dir" pod "e7337cf8-4f59-4afb-9b50-6b6c9fa4a773" (UID: "e7337cf8-4f59-4afb-9b50-6b6c9fa4a773"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:24:59.155583 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:59.155487    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7337cf8-4f59-4afb-9b50-6b6c9fa4a773-model-cache" pod "e7337cf8-4f59-4afb-9b50-6b6c9fa4a773" (UID: "e7337cf8-4f59-4afb-9b50-6b6c9fa4a773"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:24:59.155900 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:59.155662    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7337cf8-4f59-4afb-9b50-6b6c9fa4a773-home" pod "e7337cf8-4f59-4afb-9b50-6b6c9fa4a773" (UID: "e7337cf8-4f59-4afb-9b50-6b6c9fa4a773"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:24:59.164592 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:59.163023    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7337cf8-4f59-4afb-9b50-6b6c9fa4a773-dshm" pod "e7337cf8-4f59-4afb-9b50-6b6c9fa4a773" (UID: "e7337cf8-4f59-4afb-9b50-6b6c9fa4a773"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:24:59.165873 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:59.165800    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-17d809/distinct-w4-17d809-kserve-6b8f5c48cd-lhrwp"
Oct 30 22:24:59.242566 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:59.188118    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7337cf8-4f59-4afb-9b50-6b6c9fa4a773-tls-certs" pod "e7337cf8-4f59-4afb-9b50-6b6c9fa4a773" (UID: "e7337cf8-4f59-4afb-9b50-6b6c9fa4a773"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:24:59.242566 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:59.188245    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7337cf8-4f59-4afb-9b50-6b6c9fa4a773-kube-api-access-9ngwt" pod "e7337cf8-4f59-4afb-9b50-6b6c9fa4a773" (UID: "e7337cf8-4f59-4afb-9b50-6b6c9fa4a773"). InnerVolumeSpecName "kube-api-access-9ngwt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:24:59.256620 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:59.255032    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e0eca81-7db1-482b-8dd6-5b28fb89e010-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e0eca81-7db1-482b-8dd6-5b28fb89e010-tmp-dir\") pod \"1e0eca81-7db1-482b-8dd6-5b28fb89e010\" (UID: \"1e0eca81-7db1-482b-8dd6-5b28fb89e010\") "
Oct 30 22:24:59.256620 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:59.255096    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e0eca81-7db1-482b-8dd6-5b28fb89e010-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e0eca81-7db1-482b-8dd6-5b28fb89e010-model-cache\") pod \"1e0eca81-7db1-482b-8dd6-5b28fb89e010\" (UID: \"1e0eca81-7db1-482b-8dd6-5b28fb89e010\") "
Oct 30 22:24:59.256620 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:59.255144    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef5e8691-42e1-471b-ba7d-90881028eca7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef5e8691-42e1-471b-ba7d-90881028eca7-model-cache\") pod \"ef5e8691-42e1-471b-ba7d-90881028eca7\" (UID: \"ef5e8691-42e1-471b-ba7d-90881028eca7\") "
Oct 30 22:24:59.256620 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:59.255205    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e0eca81-7db1-482b-8dd6-5b28fb89e010-home\" (UniqueName: \"kubernetes.io/empty-dir/1e0eca81-7db1-482b-8dd6-5b28fb89e010-home\") pod \"1e0eca81-7db1-482b-8dd6-5b28fb89e010\" (UID: \"1e0eca81-7db1-482b-8dd6-5b28fb89e010\") "
Oct 30 22:24:59.256620 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:59.255247    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e0eca81-7db1-482b-8dd6-5b28fb89e010-tls-certs\" (UniqueName: \"kubernetes.io/secret/1e0eca81-7db1-482b-8dd6-5b28fb89e010-tls-certs\") pod \"1e0eca81-7db1-482b-8dd6-5b28fb89e010\" (UID: \"1e0eca81-7db1-482b-8dd6-5b28fb89e010\") "
Oct 30 22:24:59.256620 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:59.255305    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef5e8691-42e1-471b-ba7d-90881028eca7-kube-api-access-sz84d\" (UniqueName: \"kubernetes.io/projected/ef5e8691-42e1-471b-ba7d-90881028eca7-kube-api-access-sz84d\") pod \"ef5e8691-42e1-471b-ba7d-90881028eca7\" (UID: \"ef5e8691-42e1-471b-ba7d-90881028eca7\") "
Oct 30 22:24:59.256620 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:59.255346    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef5e8691-42e1-471b-ba7d-90881028eca7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef5e8691-42e1-471b-ba7d-90881028eca7-dshm\") pod \"ef5e8691-42e1-471b-ba7d-90881028eca7\" (UID: \"ef5e8691-42e1-471b-ba7d-90881028eca7\") "
Oct 30 22:24:59.256620 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:59.255381    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e0eca81-7db1-482b-8dd6-5b28fb89e010-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e0eca81-7db1-482b-8dd6-5b28fb89e010-dshm\") pod \"1e0eca81-7db1-482b-8dd6-5b28fb89e010\" (UID: \"1e0eca81-7db1-482b-8dd6-5b28fb89e010\") "
Oct 30 22:24:59.256620 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:59.255426    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e0eca81-7db1-482b-8dd6-5b28fb89e010-kube-api-access-vhqwr\" (UniqueName: \"kubernetes.io/projected/1e0eca81-7db1-482b-8dd6-5b28fb89e010-kube-api-access-vhqwr\") pod \"1e0eca81-7db1-482b-8dd6-5b28fb89e010\" (UID: \"1e0eca81-7db1-482b-8dd6-5b28fb89e010\") "
Oct 30 22:24:59.256620 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:59.255456    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef5e8691-42e1-471b-ba7d-90881028eca7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef5e8691-42e1-471b-ba7d-90881028eca7-tmp-dir\") pod \"ef5e8691-42e1-471b-ba7d-90881028eca7\" (UID: \"ef5e8691-42e1-471b-ba7d-90881028eca7\") "
Oct 30 22:24:59.256620 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:59.255516    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef5e8691-42e1-471b-ba7d-90881028eca7-home\" (UniqueName: \"kubernetes.io/empty-dir/ef5e8691-42e1-471b-ba7d-90881028eca7-home\") pod \"ef5e8691-42e1-471b-ba7d-90881028eca7\" (UID: \"ef5e8691-42e1-471b-ba7d-90881028eca7\") "
Oct 30 22:24:59.256620 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:59.255620    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef5e8691-42e1-471b-ba7d-90881028eca7-tls-certs\" (UniqueName: \"kubernetes.io/secret/ef5e8691-42e1-471b-ba7d-90881028eca7-tls-certs\") pod \"ef5e8691-42e1-471b-ba7d-90881028eca7\" (UID: \"ef5e8691-42e1-471b-ba7d-90881028eca7\") "
Oct 30 22:24:59.256620 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:59.256245    5061 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7337cf8-4f59-4afb-9b50-6b6c9fa4a773-dshm\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:24:59.256620 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:59.256267    5061 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7337cf8-4f59-4afb-9b50-6b6c9fa4a773-tls-certs\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:24:59.256620 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:59.256280    5061 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7337cf8-4f59-4afb-9b50-6b6c9fa4a773-home\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:24:59.256620 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:59.256293    5061 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7337cf8-4f59-4afb-9b50-6b6c9fa4a773-tmp-dir\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:24:59.256620 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:59.256307    5061 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7337cf8-4f59-4afb-9b50-6b6c9fa4a773-model-cache\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:24:59.256620 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:59.256320    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ngwt\" (UniqueName: \"kubernetes.io/projected/e7337cf8-4f59-4afb-9b50-6b6c9fa4a773-kube-api-access-9ngwt\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:24:59.258726 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:59.258695    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef5e8691-42e1-471b-ba7d-90881028eca7-model-cache" pod "ef5e8691-42e1-471b-ba7d-90881028eca7" (UID: "ef5e8691-42e1-471b-ba7d-90881028eca7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:24:59.258904 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:59.258884    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e0eca81-7db1-482b-8dd6-5b28fb89e010-tmp-dir" pod "1e0eca81-7db1-482b-8dd6-5b28fb89e010" (UID: "1e0eca81-7db1-482b-8dd6-5b28fb89e010"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:24:59.259065 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:59.259046    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e0eca81-7db1-482b-8dd6-5b28fb89e010-model-cache" pod "1e0eca81-7db1-482b-8dd6-5b28fb89e010" (UID: "1e0eca81-7db1-482b-8dd6-5b28fb89e010"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:24:59.259223 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:59.259207    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e0eca81-7db1-482b-8dd6-5b28fb89e010-home" pod "1e0eca81-7db1-482b-8dd6-5b28fb89e010" (UID: "1e0eca81-7db1-482b-8dd6-5b28fb89e010"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:24:59.259385 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:59.259368    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef5e8691-42e1-471b-ba7d-90881028eca7-tmp-dir" pod "ef5e8691-42e1-471b-ba7d-90881028eca7" (UID: "ef5e8691-42e1-471b-ba7d-90881028eca7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:24:59.259593 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:59.259573    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef5e8691-42e1-471b-ba7d-90881028eca7-home" pod "ef5e8691-42e1-471b-ba7d-90881028eca7" (UID: "ef5e8691-42e1-471b-ba7d-90881028eca7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:24:59.260511 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:59.260482    5061 generic.go:356] "Generic (PLEG): container finished" podID="1e0eca81-7db1-482b-8dd6-5b28fb89e010" containerID="81f3a64544cb556ee2539afd8521fc7525f84b418f74c42af6074580ef77e5f7" exitCode=0
Oct 30 22:24:59.260683 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:59.260662    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-17d809/premium-w4-17d809-kserve-7c9bdbbcbc-5mtmm" event={"ID":"1e0eca81-7db1-482b-8dd6-5b28fb89e010","Type":"ContainerDied","Data":"81f3a64544cb556ee2539afd8521fc7525f84b418f74c42af6074580ef77e5f7"}
Oct 30 22:24:59.260760 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:59.260703    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-17d809/premium-w4-17d809-kserve-7c9bdbbcbc-5mtmm" event={"ID":"1e0eca81-7db1-482b-8dd6-5b28fb89e010","Type":"ContainerDied","Data":"aca4ef922fa5d6485f175c0d37068ce891fb8c5983cb682152ff7e72ceb8e246"}
Oct 30 22:24:59.260760 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:59.260725    5061 scope.go:122] "RemoveContainer" containerID="81f3a64544cb556ee2539afd8521fc7525f84b418f74c42af6074580ef77e5f7"
Oct 30 22:24:59.260902 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:59.260879    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-17d809/premium-w4-17d809-kserve-7c9bdbbcbc-5mtmm"
Oct 30 22:24:59.261949 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:59.261923    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef5e8691-42e1-471b-ba7d-90881028eca7-dshm" pod "ef5e8691-42e1-471b-ba7d-90881028eca7" (UID: "ef5e8691-42e1-471b-ba7d-90881028eca7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:24:59.264078 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:59.264043    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e0eca81-7db1-482b-8dd6-5b28fb89e010-dshm" pod "1e0eca81-7db1-482b-8dd6-5b28fb89e010" (UID: "1e0eca81-7db1-482b-8dd6-5b28fb89e010"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:24:59.267734 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:59.264656    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75126b1f-hzlhv" event={"ID":"17180b54-43c4-4af7-af97-ae8e37a2ba46","Type":"ContainerDied","Data":"65edd48ca2b69b6cdaa4b9100213646b0020e36773f71e8797d4c0a6bc2daf47"}
Oct 30 22:24:59.267734 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:59.264713    5061 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="65edd48ca2b69b6cdaa4b9100213646b0020e36773f71e8797d4c0a6bc2daf47"
Oct 30 22:24:59.267734 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:59.264775    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75126b1f-hzlhv"
Oct 30 22:24:59.268001 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:59.267748    5061 generic.go:356] "Generic (PLEG): container finished" podID="fa405318-3b68-43f6-bfe4-d1fc80834e11" containerID="2baa7f9fab031ecca26095d5131691cc319d4096201108d71660d90906783439" exitCode=0
Oct 30 22:24:59.268001 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:59.267840    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rsj7x" event={"ID":"fa405318-3b68-43f6-bfe4-d1fc80834e11","Type":"ContainerDied","Data":"2baa7f9fab031ecca26095d5131691cc319d4096201108d71660d90906783439"}
Oct 30 22:24:59.268001 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:59.267878    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rsj7x" event={"ID":"fa405318-3b68-43f6-bfe4-d1fc80834e11","Type":"ContainerDied","Data":"fd2e5aec210805e68cc57d52a909d3fb37495c583877df4a5e3e84b3d00cf0c4"}
Oct 30 22:24:59.268001 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:59.267954    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rsj7x"
Oct 30 22:24:59.273629 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:59.273605    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8449ec64-vv9hj" event={"ID":"86f24e81-ee68-4d6f-b1f7-c55ff25c872d","Type":"ContainerDied","Data":"93a41795455e1b31f1ac447fad927d806a129d586a8d20ee158d59795813b152"}
Oct 30 22:24:59.273754 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:59.273650    5061 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="93a41795455e1b31f1ac447fad927d806a129d586a8d20ee158d59795813b152"
Oct 30 22:24:59.273754 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:59.273710    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8449ec64-vv9hj"
Oct 30 22:24:59.280290 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:59.280264    5061 generic.go:356] "Generic (PLEG): container finished" podID="ef5e8691-42e1-471b-ba7d-90881028eca7" containerID="a95e9b0f3ae527aeb8d1e8c8f557fc55453a836cd2574420c5a0e220bebd705a" exitCode=0
Oct 30 22:24:59.280388 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:59.280364    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-17d809/distinct-w4-17d809-kserve-6b8f5c48cd-lhrwp" event={"ID":"ef5e8691-42e1-471b-ba7d-90881028eca7","Type":"ContainerDied","Data":"a95e9b0f3ae527aeb8d1e8c8f557fc55453a836cd2574420c5a0e220bebd705a"}
Oct 30 22:24:59.280446 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:59.280390    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-17d809/distinct-w4-17d809-kserve-6b8f5c48cd-lhrwp" event={"ID":"ef5e8691-42e1-471b-ba7d-90881028eca7","Type":"ContainerDied","Data":"0b15259dff7ee7ea86d2e7b698cb27c4e2c76315a7514a64ca19ede8cf051889"}
Oct 30 22:24:59.280504 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:59.280448    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-17d809/distinct-w4-17d809-kserve-6b8f5c48cd-lhrwp"
Oct 30 22:24:59.282664 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:59.282640    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e0eca81-7db1-482b-8dd6-5b28fb89e010-tls-certs" pod "1e0eca81-7db1-482b-8dd6-5b28fb89e010" (UID: "1e0eca81-7db1-482b-8dd6-5b28fb89e010"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:24:59.283090 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:59.283068    5061 generic.go:356] "Generic (PLEG): container finished" podID="e7337cf8-4f59-4afb-9b50-6b6c9fa4a773" containerID="92fd7af56c5fb28daced7d6a9e574f44e4eed09cf90c8e4d8851bc2a78c11333" exitCode=0
Oct 30 22:24:59.283183 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:59.283169    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-17d809/simulator-w4-17d809-kserve-78d67b6f4-7hsvb" event={"ID":"e7337cf8-4f59-4afb-9b50-6b6c9fa4a773","Type":"ContainerDied","Data":"92fd7af56c5fb28daced7d6a9e574f44e4eed09cf90c8e4d8851bc2a78c11333"}
Oct 30 22:24:59.283250 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:59.283198    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-17d809/simulator-w4-17d809-kserve-78d67b6f4-7hsvb" event={"ID":"e7337cf8-4f59-4afb-9b50-6b6c9fa4a773","Type":"ContainerDied","Data":"3537553b37ce1a1823f026ffa7bc4e058173b3cb104f4591e5f77130757be781"}
Oct 30 22:24:59.283309 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:59.283250    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-17d809/simulator-w4-17d809-kserve-78d67b6f4-7hsvb"
Oct 30 22:24:59.284009 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:59.283985    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e0eca81-7db1-482b-8dd6-5b28fb89e010-kube-api-access-vhqwr" pod "1e0eca81-7db1-482b-8dd6-5b28fb89e010" (UID: "1e0eca81-7db1-482b-8dd6-5b28fb89e010"). InnerVolumeSpecName "kube-api-access-vhqwr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:24:59.284171 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:59.284151    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef5e8691-42e1-471b-ba7d-90881028eca7-kube-api-access-sz84d" pod "ef5e8691-42e1-471b-ba7d-90881028eca7" (UID: "ef5e8691-42e1-471b-ba7d-90881028eca7"). InnerVolumeSpecName "kube-api-access-sz84d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:24:59.285959 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:59.285922    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef5e8691-42e1-471b-ba7d-90881028eca7-tls-certs" pod "ef5e8691-42e1-471b-ba7d-90881028eca7" (UID: "ef5e8691-42e1-471b-ba7d-90881028eca7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:24:59.305574 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:59.305551    5061 scope.go:122] "RemoveContainer" containerID="81f3a64544cb556ee2539afd8521fc7525f84b418f74c42af6074580ef77e5f7"
Oct 30 22:24:59.307007 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:24:59.306853    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81f3a64544cb556ee2539afd8521fc7525f84b418f74c42af6074580ef77e5f7\": container with ID starting with 81f3a64544cb556ee2539afd8521fc7525f84b418f74c42af6074580ef77e5f7 not found: ID does not exist" containerID="81f3a64544cb556ee2539afd8521fc7525f84b418f74c42af6074580ef77e5f7"
Oct 30 22:24:59.307007 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:59.306893    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81f3a64544cb556ee2539afd8521fc7525f84b418f74c42af6074580ef77e5f7"} err="failed to get container status \"81f3a64544cb556ee2539afd8521fc7525f84b418f74c42af6074580ef77e5f7\": rpc error: code = NotFound desc = could not find container \"81f3a64544cb556ee2539afd8521fc7525f84b418f74c42af6074580ef77e5f7\": container with ID starting with 81f3a64544cb556ee2539afd8521fc7525f84b418f74c42af6074580ef77e5f7 not found: ID does not exist"
Oct 30 22:24:59.307007 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:59.306920    5061 scope.go:122] "RemoveContainer" containerID="2baa7f9fab031ecca26095d5131691cc319d4096201108d71660d90906783439"
Oct 30 22:24:59.337836 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:59.330393    5061 scope.go:122] "RemoveContainer" containerID="2baa7f9fab031ecca26095d5131691cc319d4096201108d71660d90906783439"
Oct 30 22:24:59.337836 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:24:59.330814    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2baa7f9fab031ecca26095d5131691cc319d4096201108d71660d90906783439\": container with ID starting with 2baa7f9fab031ecca26095d5131691cc319d4096201108d71660d90906783439 not found: ID does not exist" containerID="2baa7f9fab031ecca26095d5131691cc319d4096201108d71660d90906783439"
Oct 30 22:24:59.337836 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:59.330851    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2baa7f9fab031ecca26095d5131691cc319d4096201108d71660d90906783439"} err="failed to get container status \"2baa7f9fab031ecca26095d5131691cc319d4096201108d71660d90906783439\": rpc error: code = NotFound desc = could not find container \"2baa7f9fab031ecca26095d5131691cc319d4096201108d71660d90906783439\": container with ID starting with 2baa7f9fab031ecca26095d5131691cc319d4096201108d71660d90906783439 not found: ID does not exist"
Oct 30 22:24:59.337836 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:59.330877    5061 scope.go:122] "RemoveContainer" containerID="a95e9b0f3ae527aeb8d1e8c8f557fc55453a836cd2574420c5a0e220bebd705a"
Oct 30 22:24:59.424974 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:59.400674    5061 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e0eca81-7db1-482b-8dd6-5b28fb89e010-tmp-dir\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:24:59.424974 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:59.400703    5061 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e0eca81-7db1-482b-8dd6-5b28fb89e010-model-cache\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:24:59.424974 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:59.400718    5061 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef5e8691-42e1-471b-ba7d-90881028eca7-model-cache\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:24:59.424974 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:59.400731    5061 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e0eca81-7db1-482b-8dd6-5b28fb89e010-home\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:24:59.424974 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:59.400743    5061 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e0eca81-7db1-482b-8dd6-5b28fb89e010-tls-certs\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:24:59.424974 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:59.400771    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sz84d\" (UniqueName: \"kubernetes.io/projected/ef5e8691-42e1-471b-ba7d-90881028eca7-kube-api-access-sz84d\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:24:59.424974 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:59.400785    5061 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef5e8691-42e1-471b-ba7d-90881028eca7-dshm\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:24:59.424974 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:59.400798    5061 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e0eca81-7db1-482b-8dd6-5b28fb89e010-dshm\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:24:59.424974 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:59.400812    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhqwr\" (UniqueName: \"kubernetes.io/projected/1e0eca81-7db1-482b-8dd6-5b28fb89e010-kube-api-access-vhqwr\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:24:59.424974 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:59.400824    5061 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef5e8691-42e1-471b-ba7d-90881028eca7-tmp-dir\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:24:59.424974 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:59.400835    5061 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef5e8691-42e1-471b-ba7d-90881028eca7-home\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:24:59.424974 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:59.400848    5061 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef5e8691-42e1-471b-ba7d-90881028eca7-tls-certs\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:24:59.424974 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:59.402576    5061 scope.go:122] "RemoveContainer" containerID="a95e9b0f3ae527aeb8d1e8c8f557fc55453a836cd2574420c5a0e220bebd705a"
Oct 30 22:24:59.424974 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:24:59.402975    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a95e9b0f3ae527aeb8d1e8c8f557fc55453a836cd2574420c5a0e220bebd705a\": container with ID starting with a95e9b0f3ae527aeb8d1e8c8f557fc55453a836cd2574420c5a0e220bebd705a not found: ID does not exist" containerID="a95e9b0f3ae527aeb8d1e8c8f557fc55453a836cd2574420c5a0e220bebd705a"
Oct 30 22:24:59.424974 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:59.403009    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a95e9b0f3ae527aeb8d1e8c8f557fc55453a836cd2574420c5a0e220bebd705a"} err="failed to get container status \"a95e9b0f3ae527aeb8d1e8c8f557fc55453a836cd2574420c5a0e220bebd705a\": rpc error: code = NotFound desc = could not find container \"a95e9b0f3ae527aeb8d1e8c8f557fc55453a836cd2574420c5a0e220bebd705a\": container with ID starting with a95e9b0f3ae527aeb8d1e8c8f557fc55453a836cd2574420c5a0e220bebd705a not found: ID does not exist"
Oct 30 22:24:59.424974 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:59.403031    5061 scope.go:122] "RemoveContainer" containerID="92fd7af56c5fb28daced7d6a9e574f44e4eed09cf90c8e4d8851bc2a78c11333"
Oct 30 22:24:59.505119 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:59.504027    5061 scope.go:122] "RemoveContainer" containerID="92fd7af56c5fb28daced7d6a9e574f44e4eed09cf90c8e4d8851bc2a78c11333"
Oct 30 22:24:59.505119 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:24:59.504431    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92fd7af56c5fb28daced7d6a9e574f44e4eed09cf90c8e4d8851bc2a78c11333\": container with ID starting with 92fd7af56c5fb28daced7d6a9e574f44e4eed09cf90c8e4d8851bc2a78c11333 not found: ID does not exist" containerID="92fd7af56c5fb28daced7d6a9e574f44e4eed09cf90c8e4d8851bc2a78c11333"
Oct 30 22:24:59.505119 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:59.504465    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92fd7af56c5fb28daced7d6a9e574f44e4eed09cf90c8e4d8851bc2a78c11333"} err="failed to get container status \"92fd7af56c5fb28daced7d6a9e574f44e4eed09cf90c8e4d8851bc2a78c11333\": rpc error: code = NotFound desc = could not find container \"92fd7af56c5fb28daced7d6a9e574f44e4eed09cf90c8e4d8851bc2a78c11333\": container with ID starting with 92fd7af56c5fb28daced7d6a9e574f44e4eed09cf90c8e4d8851bc2a78c11333 not found: ID does not exist"
Oct 30 22:24:59.696518 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:59.696481    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rsj7x"]
Oct 30 22:24:59.702341 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:59.702314    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rsj7x"]
Oct 30 22:24:59.801052 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:59.801023    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-17d809/simulator-w4-17d809-kserve-78d67b6f4-7hsvb"]
Oct 30 22:24:59.804511 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:24:59.804484    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-17d809/simulator-w4-17d809-kserve-78d67b6f4-7hsvb"]
Oct 30 22:25:00.025692 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:00.019741    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-17d809/distinct-w4-17d809-kserve-6b8f5c48cd-lhrwp"]
Oct 30 22:25:00.070269 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:00.070231    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-17d809/distinct-w4-17d809-kserve-6b8f5c48cd-lhrwp"]
Oct 30 22:25:00.196867 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:00.196797    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-17d809/premium-w4-17d809-kserve-7c9bdbbcbc-5mtmm"]
Oct 30 22:25:00.201275 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:00.199231    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-17d809/premium-w4-17d809-kserve-7c9bdbbcbc-5mtmm"]
Oct 30 22:25:00.570560 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:00.569382    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e0eca81-7db1-482b-8dd6-5b28fb89e010" path="/var/lib/kubelet/pods/1e0eca81-7db1-482b-8dd6-5b28fb89e010/volumes"
Oct 30 22:25:00.570560 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:00.570087    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7337cf8-4f59-4afb-9b50-6b6c9fa4a773" path="/var/lib/kubelet/pods/e7337cf8-4f59-4afb-9b50-6b6c9fa4a773/volumes"
Oct 30 22:25:00.584962 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:00.583818    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef5e8691-42e1-471b-ba7d-90881028eca7" path="/var/lib/kubelet/pods/ef5e8691-42e1-471b-ba7d-90881028eca7/volumes"
Oct 30 22:25:00.584962 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:00.584485    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa405318-3b68-43f6-bfe4-d1fc80834e11" path="/var/lib/kubelet/pods/fa405318-3b68-43f6-bfe4-d1fc80834e11/volumes"
Oct 30 22:25:02.006717 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:02.006681    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-17d809-686d4b9688-4zrv4" podUID="71fc13bb-d3e2-4a61-9942-4195b36d227c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:25:11.815302 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:11.815209    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2d1f91-kz45z"]
Oct 30 22:25:11.861999 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:11.861969    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2d1f91-kz45z"]
Oct 30 22:25:12.006612 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:12.006579    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-17d809-686d4b9688-4zrv4" podUID="71fc13bb-d3e2-4a61-9942-4195b36d227c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:25:12.529076 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:12.529042    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd2eaa38-7e77-42e1-87c0-dd3b7e0ed0e1" path="/var/lib/kubelet/pods/dd2eaa38-7e77-42e1-87c0-dd3b7e0ed0e1/volumes"
Oct 30 22:25:16.278555 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:16.277699    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-17d809-686d4b9688-4zrv4"
Oct 30 22:25:16.474603 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:16.471271    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71fc13bb-d3e2-4a61-9942-4195b36d227c-kube-api-access-zxxjq\" (UniqueName: \"kubernetes.io/projected/71fc13bb-d3e2-4a61-9942-4195b36d227c-kube-api-access-zxxjq\") pod \"71fc13bb-d3e2-4a61-9942-4195b36d227c\" (UID: \"71fc13bb-d3e2-4a61-9942-4195b36d227c\") "
Oct 30 22:25:16.474603 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:16.471493    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71fc13bb-d3e2-4a61-9942-4195b36d227c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71fc13bb-d3e2-4a61-9942-4195b36d227c-plugins-config-volume\") pod \"71fc13bb-d3e2-4a61-9942-4195b36d227c\" (UID: \"71fc13bb-d3e2-4a61-9942-4195b36d227c\") "
Oct 30 22:25:16.474603 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:16.472686    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71fc13bb-d3e2-4a61-9942-4195b36d227c-plugins-config-volume" pod "71fc13bb-d3e2-4a61-9942-4195b36d227c" (UID: "71fc13bb-d3e2-4a61-9942-4195b36d227c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:25:16.474603 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:16.473333    5061 generic.go:356] "Generic (PLEG): container finished" podID="71fc13bb-d3e2-4a61-9942-4195b36d227c" containerID="80aa3195220ba7a419be23e77dac98eebdc4d67dde595d19e0af8354740743b9" exitCode=0
Oct 30 22:25:16.474603 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:16.473392    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-17d809-686d4b9688-4zrv4" event={"ID":"71fc13bb-d3e2-4a61-9942-4195b36d227c","Type":"ContainerDied","Data":"80aa3195220ba7a419be23e77dac98eebdc4d67dde595d19e0af8354740743b9"}
Oct 30 22:25:16.474603 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:16.473462    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-17d809-686d4b9688-4zrv4" event={"ID":"71fc13bb-d3e2-4a61-9942-4195b36d227c","Type":"ContainerDied","Data":"653a8fa7ff5281ed23a02b154b13daafcde16c7fd9279de86a0ed18b45d4e1e5"}
Oct 30 22:25:16.474603 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:16.473489    5061 scope.go:122] "RemoveContainer" containerID="80aa3195220ba7a419be23e77dac98eebdc4d67dde595d19e0af8354740743b9"
Oct 30 22:25:16.474603 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:16.473488    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-17d809-686d4b9688-4zrv4"
Oct 30 22:25:16.480055 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:16.480025    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71fc13bb-d3e2-4a61-9942-4195b36d227c-kube-api-access-zxxjq" pod "71fc13bb-d3e2-4a61-9942-4195b36d227c" (UID: "71fc13bb-d3e2-4a61-9942-4195b36d227c"). InnerVolumeSpecName "kube-api-access-zxxjq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:25:16.487542 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:16.487465    5061 scope.go:122] "RemoveContainer" containerID="80aa3195220ba7a419be23e77dac98eebdc4d67dde595d19e0af8354740743b9"
Oct 30 22:25:16.487832 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:25:16.487811    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80aa3195220ba7a419be23e77dac98eebdc4d67dde595d19e0af8354740743b9\": container with ID starting with 80aa3195220ba7a419be23e77dac98eebdc4d67dde595d19e0af8354740743b9 not found: ID does not exist" containerID="80aa3195220ba7a419be23e77dac98eebdc4d67dde595d19e0af8354740743b9"
Oct 30 22:25:16.487877 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:16.487842    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80aa3195220ba7a419be23e77dac98eebdc4d67dde595d19e0af8354740743b9"} err="failed to get container status \"80aa3195220ba7a419be23e77dac98eebdc4d67dde595d19e0af8354740743b9\": rpc error: code = NotFound desc = could not find container \"80aa3195220ba7a419be23e77dac98eebdc4d67dde595d19e0af8354740743b9\": container with ID starting with 80aa3195220ba7a419be23e77dac98eebdc4d67dde595d19e0af8354740743b9 not found: ID does not exist"
Oct 30 22:25:16.573726 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:16.573694    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxxjq\" (UniqueName: \"kubernetes.io/projected/71fc13bb-d3e2-4a61-9942-4195b36d227c-kube-api-access-zxxjq\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:25:16.573726 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:16.573720    5061 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71fc13bb-d3e2-4a61-9942-4195b36d227c-plugins-config-volume\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:25:16.889391 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:16.889354    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-17d809-686d4b9688-4zrv4"]
Oct 30 22:25:16.909555 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:16.905630    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-17d809-686d4b9688-4zrv4"]
Oct 30 22:25:18.528288 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:18.528250    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71fc13bb-d3e2-4a61-9942-4195b36d227c" path="/var/lib/kubelet/pods/71fc13bb-d3e2-4a61-9942-4195b36d227c/volumes"
Oct 30 22:25:19.654463 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:19.654383    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-mpcvw"]
Oct 30 22:25:19.654947 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:19.654602    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-mpcvw" podUID="d54aa532-ac30-469a-9d86-5b6e227e78c4" containerName="istio-proxy" containerID="cri-o://077c79aa167cffafb2f96a63f3e8416d1d139c8083321366b77da2fdf9b4f998" gracePeriod=30
Oct 30 22:25:23.105248 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:23.105216    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-edf6d5-6vblg"]
Oct 30 22:25:23.152131 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:23.152100    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-edf6d5-6vblg"]
Oct 30 22:25:24.529754 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:24.529724    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="005a7ef7-c814-4c62-9e53-21aaff9c2640" path="/var/lib/kubelet/pods/005a7ef7-c814-4c62-9e53-21aaff9c2640/volumes"
Oct 30 22:25:24.668011 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:25:24.667964    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 22:25:24.930745 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:24.930719    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-mpcvw"
Oct 30 22:25:24.979950 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:24.979882    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d54aa532-ac30-469a-9d86-5b6e227e78c4-istio-token\" (UniqueName: \"kubernetes.io/projected/d54aa532-ac30-469a-9d86-5b6e227e78c4-istio-token\") pod \"d54aa532-ac30-469a-9d86-5b6e227e78c4\" (UID: \"d54aa532-ac30-469a-9d86-5b6e227e78c4\") "
Oct 30 22:25:24.980126 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:24.980046    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d54aa532-ac30-469a-9d86-5b6e227e78c4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d54aa532-ac30-469a-9d86-5b6e227e78c4-istio-envoy\") pod \"d54aa532-ac30-469a-9d86-5b6e227e78c4\" (UID: \"d54aa532-ac30-469a-9d86-5b6e227e78c4\") "
Oct 30 22:25:24.980190 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:24.980132    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d54aa532-ac30-469a-9d86-5b6e227e78c4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d54aa532-ac30-469a-9d86-5b6e227e78c4-workload-certs\") pod \"d54aa532-ac30-469a-9d86-5b6e227e78c4\" (UID: \"d54aa532-ac30-469a-9d86-5b6e227e78c4\") "
Oct 30 22:25:24.980255 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:24.980230    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d54aa532-ac30-469a-9d86-5b6e227e78c4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d54aa532-ac30-469a-9d86-5b6e227e78c4-workload-socket\") pod \"d54aa532-ac30-469a-9d86-5b6e227e78c4\" (UID: \"d54aa532-ac30-469a-9d86-5b6e227e78c4\") "
Oct 30 22:25:24.980310 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:24.980293    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d54aa532-ac30-469a-9d86-5b6e227e78c4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d54aa532-ac30-469a-9d86-5b6e227e78c4-istio-podinfo\") pod \"d54aa532-ac30-469a-9d86-5b6e227e78c4\" (UID: \"d54aa532-ac30-469a-9d86-5b6e227e78c4\") "
Oct 30 22:25:24.980415 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:24.980387    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d54aa532-ac30-469a-9d86-5b6e227e78c4-workload-certs" pod "d54aa532-ac30-469a-9d86-5b6e227e78c4" (UID: "d54aa532-ac30-469a-9d86-5b6e227e78c4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:25:24.980481 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:24.980422    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d54aa532-ac30-469a-9d86-5b6e227e78c4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d54aa532-ac30-469a-9d86-5b6e227e78c4-istio-data\") pod \"d54aa532-ac30-469a-9d86-5b6e227e78c4\" (UID: \"d54aa532-ac30-469a-9d86-5b6e227e78c4\") "
Oct 30 22:25:24.980578 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:24.980551    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d54aa532-ac30-469a-9d86-5b6e227e78c4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d54aa532-ac30-469a-9d86-5b6e227e78c4-credential-socket\") pod \"d54aa532-ac30-469a-9d86-5b6e227e78c4\" (UID: \"d54aa532-ac30-469a-9d86-5b6e227e78c4\") "
Oct 30 22:25:24.980644 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:24.980617    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d54aa532-ac30-469a-9d86-5b6e227e78c4-workload-socket" pod "d54aa532-ac30-469a-9d86-5b6e227e78c4" (UID: "d54aa532-ac30-469a-9d86-5b6e227e78c4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:25:24.980905 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:24.980625    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d54aa532-ac30-469a-9d86-5b6e227e78c4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d54aa532-ac30-469a-9d86-5b6e227e78c4-istiod-ca-cert\") pod \"d54aa532-ac30-469a-9d86-5b6e227e78c4\" (UID: \"d54aa532-ac30-469a-9d86-5b6e227e78c4\") "
Oct 30 22:25:24.980977 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:24.980794    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d54aa532-ac30-469a-9d86-5b6e227e78c4-istio-data" pod "d54aa532-ac30-469a-9d86-5b6e227e78c4" (UID: "d54aa532-ac30-469a-9d86-5b6e227e78c4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:25:24.980977 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:24.980871    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d54aa532-ac30-469a-9d86-5b6e227e78c4-credential-socket" pod "d54aa532-ac30-469a-9d86-5b6e227e78c4" (UID: "d54aa532-ac30-469a-9d86-5b6e227e78c4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:25:24.980977 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:24.980916    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d54aa532-ac30-469a-9d86-5b6e227e78c4-istiod-ca-cert" pod "d54aa532-ac30-469a-9d86-5b6e227e78c4" (UID: "d54aa532-ac30-469a-9d86-5b6e227e78c4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:25:24.980977 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:24.980937    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d54aa532-ac30-469a-9d86-5b6e227e78c4-kube-api-access-h7ctg\" (UniqueName: \"kubernetes.io/projected/d54aa532-ac30-469a-9d86-5b6e227e78c4-kube-api-access-h7ctg\") pod \"d54aa532-ac30-469a-9d86-5b6e227e78c4\" (UID: \"d54aa532-ac30-469a-9d86-5b6e227e78c4\") "
Oct 30 22:25:24.981894 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:24.981872    5061 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d54aa532-ac30-469a-9d86-5b6e227e78c4-workload-socket\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:25:24.981966 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:24.981900    5061 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d54aa532-ac30-469a-9d86-5b6e227e78c4-istio-data\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:25:24.981966 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:24.981915    5061 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d54aa532-ac30-469a-9d86-5b6e227e78c4-credential-socket\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:25:24.981966 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:24.981930    5061 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d54aa532-ac30-469a-9d86-5b6e227e78c4-istiod-ca-cert\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:25:24.981966 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:24.981947    5061 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d54aa532-ac30-469a-9d86-5b6e227e78c4-workload-certs\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:25:24.982565 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:24.982514    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d54aa532-ac30-469a-9d86-5b6e227e78c4-istio-token" pod "d54aa532-ac30-469a-9d86-5b6e227e78c4" (UID: "d54aa532-ac30-469a-9d86-5b6e227e78c4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:25:24.982668 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:24.982651    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d54aa532-ac30-469a-9d86-5b6e227e78c4-istio-envoy" pod "d54aa532-ac30-469a-9d86-5b6e227e78c4" (UID: "d54aa532-ac30-469a-9d86-5b6e227e78c4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:25:24.982903 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:24.982875    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d54aa532-ac30-469a-9d86-5b6e227e78c4-istio-podinfo" pod "d54aa532-ac30-469a-9d86-5b6e227e78c4" (UID: "d54aa532-ac30-469a-9d86-5b6e227e78c4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:25:24.983036 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:24.983013    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d54aa532-ac30-469a-9d86-5b6e227e78c4-kube-api-access-h7ctg" pod "d54aa532-ac30-469a-9d86-5b6e227e78c4" (UID: "d54aa532-ac30-469a-9d86-5b6e227e78c4"). InnerVolumeSpecName "kube-api-access-h7ctg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:25:25.082847 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:25.082782    5061 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d54aa532-ac30-469a-9d86-5b6e227e78c4-istio-token\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:25:25.082847 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:25.082809    5061 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d54aa532-ac30-469a-9d86-5b6e227e78c4-istio-envoy\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:25:25.082847 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:25.082819    5061 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d54aa532-ac30-469a-9d86-5b6e227e78c4-istio-podinfo\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:25:25.082847 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:25.082829    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7ctg\" (UniqueName: \"kubernetes.io/projected/d54aa532-ac30-469a-9d86-5b6e227e78c4-kube-api-access-h7ctg\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:25:25.538399 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:25.538308    5061 generic.go:356] "Generic (PLEG): container finished" podID="d54aa532-ac30-469a-9d86-5b6e227e78c4" containerID="077c79aa167cffafb2f96a63f3e8416d1d139c8083321366b77da2fdf9b4f998" exitCode=0
Oct 30 22:25:25.538399 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:25.538346    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-mpcvw"
Oct 30 22:25:25.538399 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:25.538389    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-mpcvw" event={"ID":"d54aa532-ac30-469a-9d86-5b6e227e78c4","Type":"ContainerDied","Data":"077c79aa167cffafb2f96a63f3e8416d1d139c8083321366b77da2fdf9b4f998"}
Oct 30 22:25:25.539013 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:25.538434    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-mpcvw" event={"ID":"d54aa532-ac30-469a-9d86-5b6e227e78c4","Type":"ContainerDied","Data":"8074d848b729fbccf2404b685568e452a984639650ad7549d94dd7e063dd1e43"}
Oct 30 22:25:25.539013 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:25.538457    5061 scope.go:122] "RemoveContainer" containerID="077c79aa167cffafb2f96a63f3e8416d1d139c8083321366b77da2fdf9b4f998"
Oct 30 22:25:25.549791 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:25.549776    5061 scope.go:122] "RemoveContainer" containerID="077c79aa167cffafb2f96a63f3e8416d1d139c8083321366b77da2fdf9b4f998"
Oct 30 22:25:25.550048 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:25:25.550031    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"077c79aa167cffafb2f96a63f3e8416d1d139c8083321366b77da2fdf9b4f998\": container with ID starting with 077c79aa167cffafb2f96a63f3e8416d1d139c8083321366b77da2fdf9b4f998 not found: ID does not exist" containerID="077c79aa167cffafb2f96a63f3e8416d1d139c8083321366b77da2fdf9b4f998"
Oct 30 22:25:25.550099 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:25.550055    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"077c79aa167cffafb2f96a63f3e8416d1d139c8083321366b77da2fdf9b4f998"} err="failed to get container status \"077c79aa167cffafb2f96a63f3e8416d1d139c8083321366b77da2fdf9b4f998\": rpc error: code = NotFound desc = could not find container \"077c79aa167cffafb2f96a63f3e8416d1d139c8083321366b77da2fdf9b4f998\": container with ID starting with 077c79aa167cffafb2f96a63f3e8416d1d139c8083321366b77da2fdf9b4f998 not found: ID does not exist"
Oct 30 22:25:25.696492 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:25.696461    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-mpcvw"]
Oct 30 22:25:25.743293 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:25.743265    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-mpcvw"]
Oct 30 22:25:26.528305 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:26.528274    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d54aa532-ac30-469a-9d86-5b6e227e78c4" path="/var/lib/kubelet/pods/d54aa532-ac30-469a-9d86-5b6e227e78c4/volumes"
Oct 30 22:25:28.970462 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:28.970428    5061 scope.go:122] "RemoveContainer" containerID="1772ccf2861a0b213f8e446a375c35a4eae685b64fa0d5e62ad46a77423657dc"
Oct 30 22:25:31.293028 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:31.292992    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c1e2ab-28wvf"]
Oct 30 22:25:31.293668 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:31.293648    5061 cpu_manager.go:414] "containerMap: removing container" podUID="d54aa532-ac30-469a-9d86-5b6e227e78c4" containerName="istio-proxy"
Oct 30 22:25:31.293668 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:31.293668    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d54aa532-ac30-469a-9d86-5b6e227e78c4" containerName="istio-proxy"
Oct 30 22:25:31.293819 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:31.293680    5061 cpu_manager.go:414] "containerMap: removing container" podUID="ef5e8691-42e1-471b-ba7d-90881028eca7" containerName="main"
Oct 30 22:25:31.293819 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:31.293686    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef5e8691-42e1-471b-ba7d-90881028eca7" containerName="main"
Oct 30 22:25:31.293819 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:31.293707    5061 cpu_manager.go:414] "containerMap: removing container" podUID="86f24e81-ee68-4d6f-b1f7-c55ff25c872d" containerName="revoke-keys"
Oct 30 22:25:31.293819 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:31.293713    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86f24e81-ee68-4d6f-b1f7-c55ff25c872d" containerName="revoke-keys"
Oct 30 22:25:31.293819 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:31.293722    5061 cpu_manager.go:414] "containerMap: removing container" podUID="e7337cf8-4f59-4afb-9b50-6b6c9fa4a773" containerName="main"
Oct 30 22:25:31.293819 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:31.293727    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7337cf8-4f59-4afb-9b50-6b6c9fa4a773" containerName="main"
Oct 30 22:25:31.293819 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:31.293738    5061 cpu_manager.go:414] "containerMap: removing container" podUID="fa405318-3b68-43f6-bfe4-d1fc80834e11" containerName="istio-proxy"
Oct 30 22:25:31.293819 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:31.293743    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa405318-3b68-43f6-bfe4-d1fc80834e11" containerName="istio-proxy"
Oct 30 22:25:31.293819 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:31.293758    5061 cpu_manager.go:414] "containerMap: removing container" podUID="17180b54-43c4-4af7-af97-ae8e37a2ba46" containerName="revoke-keys"
Oct 30 22:25:31.293819 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:31.293763    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17180b54-43c4-4af7-af97-ae8e37a2ba46" containerName="revoke-keys"
Oct 30 22:25:31.293819 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:31.293772    5061 cpu_manager.go:414] "containerMap: removing container" podUID="1e0eca81-7db1-482b-8dd6-5b28fb89e010" containerName="main"
Oct 30 22:25:31.293819 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:31.293776    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e0eca81-7db1-482b-8dd6-5b28fb89e010" containerName="main"
Oct 30 22:25:31.293819 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:31.293784    5061 cpu_manager.go:414] "containerMap: removing container" podUID="71fc13bb-d3e2-4a61-9942-4195b36d227c" containerName="payload-processing"
Oct 30 22:25:31.293819 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:31.293788    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71fc13bb-d3e2-4a61-9942-4195b36d227c" containerName="payload-processing"
Oct 30 22:25:31.294299 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:31.293871    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="d54aa532-ac30-469a-9d86-5b6e227e78c4" containerName="istio-proxy"
Oct 30 22:25:31.294299 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:31.293881    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef5e8691-42e1-471b-ba7d-90881028eca7" containerName="main"
Oct 30 22:25:31.294299 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:31.293890    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="86f24e81-ee68-4d6f-b1f7-c55ff25c872d" containerName="revoke-keys"
Oct 30 22:25:31.294299 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:31.293898    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7337cf8-4f59-4afb-9b50-6b6c9fa4a773" containerName="main"
Oct 30 22:25:31.294299 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:31.293905    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa405318-3b68-43f6-bfe4-d1fc80834e11" containerName="istio-proxy"
Oct 30 22:25:31.294299 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:31.293913    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="17180b54-43c4-4af7-af97-ae8e37a2ba46" containerName="revoke-keys"
Oct 30 22:25:31.294299 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:31.293919    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e0eca81-7db1-482b-8dd6-5b28fb89e010" containerName="main"
Oct 30 22:25:31.294299 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:31.293925    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="71fc13bb-d3e2-4a61-9942-4195b36d227c" containerName="payload-processing"
Oct 30 22:25:31.299169 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:31.299149    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c1e2ab-28wvf"
Oct 30 22:25:31.350770 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:31.350744    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:25:31.350959 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:31.350774    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gdmfg\""
Oct 30 22:25:31.368325 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:31.368296    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e9f1392-e1ff-4491-af70-c0f0987ff9b9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-c1e2ab-28wvf\" (UID: \"5e9f1392-e1ff-4491-af70-c0f0987ff9b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c1e2ab-28wvf"
Oct 30 22:25:31.368486 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:31.368436    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zp4cz\" (UniqueName: \"kubernetes.io/projected/5e9f1392-e1ff-4491-af70-c0f0987ff9b9-kube-api-access-zp4cz\") pod \"maas-api-revoke-keys-e2e-api-b-c1e2ab-28wvf\" (UID: \"5e9f1392-e1ff-4491-af70-c0f0987ff9b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c1e2ab-28wvf"
Oct 30 22:25:31.368587 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:31.368570    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e9f1392-e1ff-4491-af70-c0f0987ff9b9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-c1e2ab-28wvf\" (UID: \"5e9f1392-e1ff-4491-af70-c0f0987ff9b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c1e2ab-28wvf"
Oct 30 22:25:31.401353 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:31.401323    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c1e2ab-28wvf"]
Oct 30 22:25:31.469477 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:31.469449    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e9f1392-e1ff-4491-af70-c0f0987ff9b9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-c1e2ab-28wvf\" (UID: \"5e9f1392-e1ff-4491-af70-c0f0987ff9b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c1e2ab-28wvf"
Oct 30 22:25:31.469641 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:31.469505    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e9f1392-e1ff-4491-af70-c0f0987ff9b9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-c1e2ab-28wvf\" (UID: \"5e9f1392-e1ff-4491-af70-c0f0987ff9b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c1e2ab-28wvf"
Oct 30 22:25:31.469641 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:31.469604    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zp4cz\" (UniqueName: \"kubernetes.io/projected/5e9f1392-e1ff-4491-af70-c0f0987ff9b9-kube-api-access-zp4cz\") pod \"maas-api-revoke-keys-e2e-api-b-c1e2ab-28wvf\" (UID: \"5e9f1392-e1ff-4491-af70-c0f0987ff9b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c1e2ab-28wvf"
Oct 30 22:25:31.470178 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:31.470159    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e9f1392-e1ff-4491-af70-c0f0987ff9b9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-c1e2ab-28wvf\" (UID: \"5e9f1392-e1ff-4491-af70-c0f0987ff9b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c1e2ab-28wvf"
Oct 30 22:25:31.471774 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:31.471754    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e9f1392-e1ff-4491-af70-c0f0987ff9b9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-c1e2ab-28wvf\" (UID: \"5e9f1392-e1ff-4491-af70-c0f0987ff9b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c1e2ab-28wvf"
Oct 30 22:25:31.523542 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:31.523504    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zp4cz\" (UniqueName: \"kubernetes.io/projected/5e9f1392-e1ff-4491-af70-c0f0987ff9b9-kube-api-access-zp4cz\") pod \"maas-api-revoke-keys-e2e-api-b-c1e2ab-28wvf\" (UID: \"5e9f1392-e1ff-4491-af70-c0f0987ff9b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c1e2ab-28wvf"
Oct 30 22:25:31.611965 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:31.611940    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c1e2ab-28wvf"
Oct 30 22:25:31.980929 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:31.980892    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c1e2ab-28wvf"]
Oct 30 22:25:31.981931 ip-10-0-130-60 kubenswrapper[5061]: W1030 22:25:31.981902    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e9f1392_e1ff_4491_af70_c0f0987ff9b9.slice/crio-0b4652e7a847a3ec818a39d8b1511735754890b485846cec346e465791c6bd5f WatchSource:0}: Error finding container 0b4652e7a847a3ec818a39d8b1511735754890b485846cec346e465791c6bd5f: Status 404 returned error can't find the container with id 0b4652e7a847a3ec818a39d8b1511735754890b485846cec346e465791c6bd5f
Oct 30 22:25:32.596386 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:32.596309    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c1e2ab-28wvf" event={"ID":"5e9f1392-e1ff-4491-af70-c0f0987ff9b9","Type":"ContainerStarted","Data":"b184ee3e93d46255d09ef818e3c251e1de74d3af184c436e4ee449355b3802ab"}
Oct 30 22:25:32.596386 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:32.596361    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c1e2ab-28wvf" event={"ID":"5e9f1392-e1ff-4491-af70-c0f0987ff9b9","Type":"ContainerStarted","Data":"0b4652e7a847a3ec818a39d8b1511735754890b485846cec346e465791c6bd5f"}
Oct 30 22:25:32.704480 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:32.704430    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c1e2ab-28wvf" podStartSLOduration=1.704417232 podStartE2EDuration="1.704417232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:25:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:25:32.703912189 +0000 UTC m=+2586.316681947" watchObservedRunningTime="2026-10-30 22:25:32.704417232 +0000 UTC m=+2586.317186990"
Oct 30 22:25:33.609153 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:33.609111    5061 generic.go:356] "Generic (PLEG): container finished" podID="5e9f1392-e1ff-4491-af70-c0f0987ff9b9" containerID="b184ee3e93d46255d09ef818e3c251e1de74d3af184c436e4ee449355b3802ab" exitCode=0
Oct 30 22:25:33.609662 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:33.609212    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c1e2ab-28wvf" event={"ID":"5e9f1392-e1ff-4491-af70-c0f0987ff9b9","Type":"ContainerDied","Data":"b184ee3e93d46255d09ef818e3c251e1de74d3af184c436e4ee449355b3802ab"}
Oct 30 22:25:34.780860 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:34.780833    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c1e2ab-28wvf"
Oct 30 22:25:34.911313 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:34.911222    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e9f1392-e1ff-4491-af70-c0f0987ff9b9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e9f1392-e1ff-4491-af70-c0f0987ff9b9-maas-api-openshift-ca\") pod \"5e9f1392-e1ff-4491-af70-c0f0987ff9b9\" (UID: \"5e9f1392-e1ff-4491-af70-c0f0987ff9b9\") "
Oct 30 22:25:34.911497 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:34.911334    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e9f1392-e1ff-4491-af70-c0f0987ff9b9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e9f1392-e1ff-4491-af70-c0f0987ff9b9-maas-api-xks-ca\") pod \"5e9f1392-e1ff-4491-af70-c0f0987ff9b9\" (UID: \"5e9f1392-e1ff-4491-af70-c0f0987ff9b9\") "
Oct 30 22:25:34.911497 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:34.911437    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e9f1392-e1ff-4491-af70-c0f0987ff9b9-kube-api-access-zp4cz\" (UniqueName: \"kubernetes.io/projected/5e9f1392-e1ff-4491-af70-c0f0987ff9b9-kube-api-access-zp4cz\") pod \"5e9f1392-e1ff-4491-af70-c0f0987ff9b9\" (UID: \"5e9f1392-e1ff-4491-af70-c0f0987ff9b9\") "
Oct 30 22:25:34.911721 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:34.911681    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e9f1392-e1ff-4491-af70-c0f0987ff9b9-maas-api-openshift-ca" pod "5e9f1392-e1ff-4491-af70-c0f0987ff9b9" (UID: "5e9f1392-e1ff-4491-af70-c0f0987ff9b9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:25:34.911982 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:34.911959    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e9f1392-e1ff-4491-af70-c0f0987ff9b9-maas-api-openshift-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:25:34.914014 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:34.913988    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e9f1392-e1ff-4491-af70-c0f0987ff9b9-kube-api-access-zp4cz" pod "5e9f1392-e1ff-4491-af70-c0f0987ff9b9" (UID: "5e9f1392-e1ff-4491-af70-c0f0987ff9b9"). InnerVolumeSpecName "kube-api-access-zp4cz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:25:34.914272 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:34.914251    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e9f1392-e1ff-4491-af70-c0f0987ff9b9-maas-api-xks-ca" pod "5e9f1392-e1ff-4491-af70-c0f0987ff9b9" (UID: "5e9f1392-e1ff-4491-af70-c0f0987ff9b9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:25:35.012887 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:35.012857    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zp4cz\" (UniqueName: \"kubernetes.io/projected/5e9f1392-e1ff-4491-af70-c0f0987ff9b9-kube-api-access-zp4cz\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:25:35.012887 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:35.012886    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e9f1392-e1ff-4491-af70-c0f0987ff9b9-maas-api-xks-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:25:35.625873 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:35.625841    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c1e2ab-28wvf"
Oct 30 22:25:35.626048 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:35.625871    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c1e2ab-28wvf" event={"ID":"5e9f1392-e1ff-4491-af70-c0f0987ff9b9","Type":"ContainerDied","Data":"0b4652e7a847a3ec818a39d8b1511735754890b485846cec346e465791c6bd5f"}
Oct 30 22:25:35.626048 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:35.625921    5061 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0b4652e7a847a3ec818a39d8b1511735754890b485846cec346e465791c6bd5f"
Oct 30 22:25:42.836754 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:42.836717    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c1e2ab-5b47865787-wj8g9"]
Oct 30 22:25:44.484219 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:44.484181    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-17d809-vqztk"]
Oct 30 22:25:44.530428 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:44.530393    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-17d809-vqztk"]
Oct 30 22:25:46.528043 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:46.528011    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b79a5098-150c-48f1-a156-e54615c1be82" path="/var/lib/kubelet/pods/b79a5098-150c-48f1-a156-e54615c1be82/volumes"
Oct 30 22:25:47.838344 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:47.838307    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c1e2ab-5b47865787-wj8g9" podUID="136b259c-03a5-4cc9-8702-fd78a552bb65" containerName="payload-pre-processing" containerID="cri-o://e4993a4a9426eaf27fda7cf07b650517de53f9947aef942d328a84b6bf9faa89" gracePeriod=25
Oct 30 22:25:48.109918 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:48.109887    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c1e2ab-5b47865787-wj8g9"
Oct 30 22:25:48.172677 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:48.172646    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/136b259c-03a5-4cc9-8702-fd78a552bb65-kube-api-access-dpjj7\" (UniqueName: \"kubernetes.io/projected/136b259c-03a5-4cc9-8702-fd78a552bb65-kube-api-access-dpjj7\") pod \"136b259c-03a5-4cc9-8702-fd78a552bb65\" (UID: \"136b259c-03a5-4cc9-8702-fd78a552bb65\") "
Oct 30 22:25:48.172866 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:48.172848    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/136b259c-03a5-4cc9-8702-fd78a552bb65-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/136b259c-03a5-4cc9-8702-fd78a552bb65-plugins-config-volume\") pod \"136b259c-03a5-4cc9-8702-fd78a552bb65\" (UID: \"136b259c-03a5-4cc9-8702-fd78a552bb65\") "
Oct 30 22:25:48.173224 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:48.173202    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/136b259c-03a5-4cc9-8702-fd78a552bb65-plugins-config-volume" pod "136b259c-03a5-4cc9-8702-fd78a552bb65" (UID: "136b259c-03a5-4cc9-8702-fd78a552bb65"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:25:48.174742 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:48.174716    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/136b259c-03a5-4cc9-8702-fd78a552bb65-kube-api-access-dpjj7" pod "136b259c-03a5-4cc9-8702-fd78a552bb65" (UID: "136b259c-03a5-4cc9-8702-fd78a552bb65"). InnerVolumeSpecName "kube-api-access-dpjj7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:25:48.274456 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:48.274427    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dpjj7\" (UniqueName: \"kubernetes.io/projected/136b259c-03a5-4cc9-8702-fd78a552bb65-kube-api-access-dpjj7\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:25:48.274456 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:48.274451    5061 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/136b259c-03a5-4cc9-8702-fd78a552bb65-plugins-config-volume\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:25:48.714266 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:48.714234    5061 generic.go:356] "Generic (PLEG): container finished" podID="136b259c-03a5-4cc9-8702-fd78a552bb65" containerID="e4993a4a9426eaf27fda7cf07b650517de53f9947aef942d328a84b6bf9faa89" exitCode=0
Oct 30 22:25:48.714454 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:48.714281    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c1e2ab-5b47865787-wj8g9"
Oct 30 22:25:48.714454 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:48.714324    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c1e2ab-5b47865787-wj8g9" event={"ID":"136b259c-03a5-4cc9-8702-fd78a552bb65","Type":"ContainerDied","Data":"e4993a4a9426eaf27fda7cf07b650517de53f9947aef942d328a84b6bf9faa89"}
Oct 30 22:25:48.714454 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:48.714374    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c1e2ab-5b47865787-wj8g9" event={"ID":"136b259c-03a5-4cc9-8702-fd78a552bb65","Type":"ContainerDied","Data":"b8c734a100f5ac7f93d999869b15c0a78cee8ed886b279a6af6c8a2cd76795f2"}
Oct 30 22:25:48.714454 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:48.714393    5061 scope.go:122] "RemoveContainer" containerID="e4993a4a9426eaf27fda7cf07b650517de53f9947aef942d328a84b6bf9faa89"
Oct 30 22:25:48.724488 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:48.724470    5061 scope.go:122] "RemoveContainer" containerID="e4993a4a9426eaf27fda7cf07b650517de53f9947aef942d328a84b6bf9faa89"
Oct 30 22:25:48.724767 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:25:48.724745    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4993a4a9426eaf27fda7cf07b650517de53f9947aef942d328a84b6bf9faa89\": container with ID starting with e4993a4a9426eaf27fda7cf07b650517de53f9947aef942d328a84b6bf9faa89 not found: ID does not exist" containerID="e4993a4a9426eaf27fda7cf07b650517de53f9947aef942d328a84b6bf9faa89"
Oct 30 22:25:48.724830 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:48.724775    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4993a4a9426eaf27fda7cf07b650517de53f9947aef942d328a84b6bf9faa89"} err="failed to get container status \"e4993a4a9426eaf27fda7cf07b650517de53f9947aef942d328a84b6bf9faa89\": rpc error: code = NotFound desc = could not find container \"e4993a4a9426eaf27fda7cf07b650517de53f9947aef942d328a84b6bf9faa89\": container with ID starting with e4993a4a9426eaf27fda7cf07b650517de53f9947aef942d328a84b6bf9faa89 not found: ID does not exist"
Oct 30 22:25:48.869074 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:48.869041    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c1e2ab-5b47865787-wj8g9"]
Oct 30 22:25:48.872273 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:48.872247    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c1e2ab-5b47865787-wj8g9"]
Oct 30 22:25:50.528689 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:50.528656    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="136b259c-03a5-4cc9-8702-fd78a552bb65" path="/var/lib/kubelet/pods/136b259c-03a5-4cc9-8702-fd78a552bb65/volumes"
Oct 30 22:25:51.481307 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:51.481273    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-ddh7p"]
Oct 30 22:25:51.481540 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:51.481504    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-ddh7p" podUID="ba3aed70-33f3-4f0f-87cd-a8c19c347cd8" containerName="istio-proxy" containerID="cri-o://680c40c3f9c87dea6ccec4aa32b8b20a014bb7104446bc4e25cf99ca1e61ea06" gracePeriod=30
Oct 30 22:25:54.046359 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:54.046325    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bef575f8-5sfmq"]
Oct 30 22:25:54.093432 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:54.093397    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bef575f8-5sfmq"]
Oct 30 22:25:54.528881 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:54.528808    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4402ea44-8969-45e8-8037-593633026e9e" path="/var/lib/kubelet/pods/4402ea44-8969-45e8-8037-593633026e9e/volumes"
Oct 30 22:25:54.667236 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:25:54.667200    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 22:25:56.754203 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:56.754144    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-ddh7p"
Oct 30 22:25:56.769177 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:56.769145    5061 generic.go:356] "Generic (PLEG): container finished" podID="ba3aed70-33f3-4f0f-87cd-a8c19c347cd8" containerID="680c40c3f9c87dea6ccec4aa32b8b20a014bb7104446bc4e25cf99ca1e61ea06" exitCode=0
Oct 30 22:25:56.769322 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:56.769191    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-ddh7p"
Oct 30 22:25:56.769322 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:56.769191    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-ddh7p" event={"ID":"ba3aed70-33f3-4f0f-87cd-a8c19c347cd8","Type":"ContainerDied","Data":"680c40c3f9c87dea6ccec4aa32b8b20a014bb7104446bc4e25cf99ca1e61ea06"}
Oct 30 22:25:56.769449 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:56.769326    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-ddh7p" event={"ID":"ba3aed70-33f3-4f0f-87cd-a8c19c347cd8","Type":"ContainerDied","Data":"0f158d80b547a3963e47a1e8f8746f9889c64e9a4ed8d0d8c9e2d6b78d7fbc54"}
Oct 30 22:25:56.769449 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:56.769344    5061 scope.go:122] "RemoveContainer" containerID="680c40c3f9c87dea6ccec4aa32b8b20a014bb7104446bc4e25cf99ca1e61ea06"
Oct 30 22:25:56.783050 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:56.782705    5061 scope.go:122] "RemoveContainer" containerID="40776579af3656e34fb53d026a34a050339786ec5e6583ede8f3d72cf8844b91"
Oct 30 22:25:56.792045 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:56.792028    5061 scope.go:122] "RemoveContainer" containerID="680c40c3f9c87dea6ccec4aa32b8b20a014bb7104446bc4e25cf99ca1e61ea06"
Oct 30 22:25:56.792292 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:25:56.792274    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"680c40c3f9c87dea6ccec4aa32b8b20a014bb7104446bc4e25cf99ca1e61ea06\": container with ID starting with 680c40c3f9c87dea6ccec4aa32b8b20a014bb7104446bc4e25cf99ca1e61ea06 not found: ID does not exist" containerID="680c40c3f9c87dea6ccec4aa32b8b20a014bb7104446bc4e25cf99ca1e61ea06"
Oct 30 22:25:56.792369 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:56.792298    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"680c40c3f9c87dea6ccec4aa32b8b20a014bb7104446bc4e25cf99ca1e61ea06"} err="failed to get container status \"680c40c3f9c87dea6ccec4aa32b8b20a014bb7104446bc4e25cf99ca1e61ea06\": rpc error: code = NotFound desc = could not find container \"680c40c3f9c87dea6ccec4aa32b8b20a014bb7104446bc4e25cf99ca1e61ea06\": container with ID starting with 680c40c3f9c87dea6ccec4aa32b8b20a014bb7104446bc4e25cf99ca1e61ea06 not found: ID does not exist"
Oct 30 22:25:56.792369 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:56.792314    5061 scope.go:122] "RemoveContainer" containerID="40776579af3656e34fb53d026a34a050339786ec5e6583ede8f3d72cf8844b91"
Oct 30 22:25:56.792589 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:25:56.792566    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40776579af3656e34fb53d026a34a050339786ec5e6583ede8f3d72cf8844b91\": container with ID starting with 40776579af3656e34fb53d026a34a050339786ec5e6583ede8f3d72cf8844b91 not found: ID does not exist" containerID="40776579af3656e34fb53d026a34a050339786ec5e6583ede8f3d72cf8844b91"
Oct 30 22:25:56.792648 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:56.792596    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40776579af3656e34fb53d026a34a050339786ec5e6583ede8f3d72cf8844b91"} err="failed to get container status \"40776579af3656e34fb53d026a34a050339786ec5e6583ede8f3d72cf8844b91\": rpc error: code = NotFound desc = could not find container \"40776579af3656e34fb53d026a34a050339786ec5e6583ede8f3d72cf8844b91\": container with ID starting with 40776579af3656e34fb53d026a34a050339786ec5e6583ede8f3d72cf8844b91 not found: ID does not exist"
Oct 30 22:25:56.873049 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:56.873023    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba3aed70-33f3-4f0f-87cd-a8c19c347cd8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba3aed70-33f3-4f0f-87cd-a8c19c347cd8-workload-certs\") pod \"ba3aed70-33f3-4f0f-87cd-a8c19c347cd8\" (UID: \"ba3aed70-33f3-4f0f-87cd-a8c19c347cd8\") "
Oct 30 22:25:56.873146 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:56.873078    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba3aed70-33f3-4f0f-87cd-a8c19c347cd8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba3aed70-33f3-4f0f-87cd-a8c19c347cd8-istio-data\") pod \"ba3aed70-33f3-4f0f-87cd-a8c19c347cd8\" (UID: \"ba3aed70-33f3-4f0f-87cd-a8c19c347cd8\") "
Oct 30 22:25:56.873146 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:56.873098    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba3aed70-33f3-4f0f-87cd-a8c19c347cd8-istio-token\" (UniqueName: \"kubernetes.io/projected/ba3aed70-33f3-4f0f-87cd-a8c19c347cd8-istio-token\") pod \"ba3aed70-33f3-4f0f-87cd-a8c19c347cd8\" (UID: \"ba3aed70-33f3-4f0f-87cd-a8c19c347cd8\") "
Oct 30 22:25:56.873224 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:56.873159    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba3aed70-33f3-4f0f-87cd-a8c19c347cd8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba3aed70-33f3-4f0f-87cd-a8c19c347cd8-istio-envoy\") pod \"ba3aed70-33f3-4f0f-87cd-a8c19c347cd8\" (UID: \"ba3aed70-33f3-4f0f-87cd-a8c19c347cd8\") "
Oct 30 22:25:56.873224 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:56.873204    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba3aed70-33f3-4f0f-87cd-a8c19c347cd8-workload-certs" pod "ba3aed70-33f3-4f0f-87cd-a8c19c347cd8" (UID: "ba3aed70-33f3-4f0f-87cd-a8c19c347cd8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:25:56.873307 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:56.873235    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba3aed70-33f3-4f0f-87cd-a8c19c347cd8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba3aed70-33f3-4f0f-87cd-a8c19c347cd8-workload-socket\") pod \"ba3aed70-33f3-4f0f-87cd-a8c19c347cd8\" (UID: \"ba3aed70-33f3-4f0f-87cd-a8c19c347cd8\") "
Oct 30 22:25:56.873307 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:56.873278    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ba3aed70-33f3-4f0f-87cd-a8c19c347cd8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba3aed70-33f3-4f0f-87cd-a8c19c347cd8-istio-podinfo\") pod \"ba3aed70-33f3-4f0f-87cd-a8c19c347cd8\" (UID: \"ba3aed70-33f3-4f0f-87cd-a8c19c347cd8\") "
Oct 30 22:25:56.873401 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:56.873304    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba3aed70-33f3-4f0f-87cd-a8c19c347cd8-kube-api-access-7jnbl\" (UniqueName: \"kubernetes.io/projected/ba3aed70-33f3-4f0f-87cd-a8c19c347cd8-kube-api-access-7jnbl\") pod \"ba3aed70-33f3-4f0f-87cd-a8c19c347cd8\" (UID: \"ba3aed70-33f3-4f0f-87cd-a8c19c347cd8\") "
Oct 30 22:25:56.873401 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:56.873313    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba3aed70-33f3-4f0f-87cd-a8c19c347cd8-istio-data" pod "ba3aed70-33f3-4f0f-87cd-a8c19c347cd8" (UID: "ba3aed70-33f3-4f0f-87cd-a8c19c347cd8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:25:56.873401 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:56.873334    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba3aed70-33f3-4f0f-87cd-a8c19c347cd8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba3aed70-33f3-4f0f-87cd-a8c19c347cd8-credential-socket\") pod \"ba3aed70-33f3-4f0f-87cd-a8c19c347cd8\" (UID: \"ba3aed70-33f3-4f0f-87cd-a8c19c347cd8\") "
Oct 30 22:25:56.873590 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:56.873444    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba3aed70-33f3-4f0f-87cd-a8c19c347cd8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba3aed70-33f3-4f0f-87cd-a8c19c347cd8-istiod-ca-cert\") pod \"ba3aed70-33f3-4f0f-87cd-a8c19c347cd8\" (UID: \"ba3aed70-33f3-4f0f-87cd-a8c19c347cd8\") "
Oct 30 22:25:56.873590 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:56.873493    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba3aed70-33f3-4f0f-87cd-a8c19c347cd8-workload-socket" pod "ba3aed70-33f3-4f0f-87cd-a8c19c347cd8" (UID: "ba3aed70-33f3-4f0f-87cd-a8c19c347cd8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:25:56.873773 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:56.873735    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba3aed70-33f3-4f0f-87cd-a8c19c347cd8-credential-socket" pod "ba3aed70-33f3-4f0f-87cd-a8c19c347cd8" (UID: "ba3aed70-33f3-4f0f-87cd-a8c19c347cd8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:25:56.873852 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:56.873833    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba3aed70-33f3-4f0f-87cd-a8c19c347cd8-istiod-ca-cert" pod "ba3aed70-33f3-4f0f-87cd-a8c19c347cd8" (UID: "ba3aed70-33f3-4f0f-87cd-a8c19c347cd8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:25:56.873994 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:56.873976    5061 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba3aed70-33f3-4f0f-87cd-a8c19c347cd8-workload-certs\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:25:56.874073 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:56.873996    5061 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba3aed70-33f3-4f0f-87cd-a8c19c347cd8-istio-data\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:25:56.874073 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:56.874010    5061 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba3aed70-33f3-4f0f-87cd-a8c19c347cd8-workload-socket\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:25:56.874073 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:56.874025    5061 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba3aed70-33f3-4f0f-87cd-a8c19c347cd8-credential-socket\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:25:56.874073 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:56.874035    5061 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba3aed70-33f3-4f0f-87cd-a8c19c347cd8-istiod-ca-cert\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:25:56.875360 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:56.875337    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba3aed70-33f3-4f0f-87cd-a8c19c347cd8-istio-token" pod "ba3aed70-33f3-4f0f-87cd-a8c19c347cd8" (UID: "ba3aed70-33f3-4f0f-87cd-a8c19c347cd8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:25:56.875488 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:56.875398    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba3aed70-33f3-4f0f-87cd-a8c19c347cd8-kube-api-access-7jnbl" pod "ba3aed70-33f3-4f0f-87cd-a8c19c347cd8" (UID: "ba3aed70-33f3-4f0f-87cd-a8c19c347cd8"). InnerVolumeSpecName "kube-api-access-7jnbl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:25:56.875793 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:56.875770    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ba3aed70-33f3-4f0f-87cd-a8c19c347cd8-istio-podinfo" pod "ba3aed70-33f3-4f0f-87cd-a8c19c347cd8" (UID: "ba3aed70-33f3-4f0f-87cd-a8c19c347cd8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:25:56.875793 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:56.875784    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba3aed70-33f3-4f0f-87cd-a8c19c347cd8-istio-envoy" pod "ba3aed70-33f3-4f0f-87cd-a8c19c347cd8" (UID: "ba3aed70-33f3-4f0f-87cd-a8c19c347cd8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:25:56.974896 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:56.974862    5061 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba3aed70-33f3-4f0f-87cd-a8c19c347cd8-istio-token\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:25:56.974896 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:56.974890    5061 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba3aed70-33f3-4f0f-87cd-a8c19c347cd8-istio-envoy\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:25:56.974896 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:56.974900    5061 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba3aed70-33f3-4f0f-87cd-a8c19c347cd8-istio-podinfo\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:25:56.975113 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:56.974909    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jnbl\" (UniqueName: \"kubernetes.io/projected/ba3aed70-33f3-4f0f-87cd-a8c19c347cd8-kube-api-access-7jnbl\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:25:57.191734 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:57.191698    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-ddh7p"]
Oct 30 22:25:57.196120 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:57.196091    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-ddh7p"]
Oct 30 22:25:58.527795 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:25:58.527757    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba3aed70-33f3-4f0f-87cd-a8c19c347cd8" path="/var/lib/kubelet/pods/ba3aed70-33f3-4f0f-87cd-a8c19c347cd8/volumes"
Oct 30 22:26:24.667083 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:26:24.667034    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 22:26:29.027835 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:26:29.027803    5061 scope.go:122] "RemoveContainer" containerID="739603148ec04cb2bcaf124f97772fe53f62a406823867d81527d53336bdbd90"
Oct 30 22:26:29.037634 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:26:29.037613    5061 scope.go:122] "RemoveContainer" containerID="105e25a4dfd53227a13fdb06989c601395d72f957f744fabdbe7f62be6c46283"
Oct 30 22:26:29.046695 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:26:29.046671    5061 scope.go:122] "RemoveContainer" containerID="f43e71ca1cab8f496e9a514e74a35faa45f1915b1543b00ea13992e68d9a2eea"
Oct 30 22:26:53.496789 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:26:53.496709    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c1e2ab-28wvf"]
Oct 30 22:26:53.543651 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:26:53.543624    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c1e2ab-28wvf"]
Oct 30 22:26:54.528789 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:26:54.528757    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e9f1392-e1ff-4491-af70-c0f0987ff9b9" path="/var/lib/kubelet/pods/5e9f1392-e1ff-4491-af70-c0f0987ff9b9/volumes"
Oct 30 22:26:54.666749 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:26:54.666712    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 22:27:00.562221 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:00.562188    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-c1e2ab-openshift-default-bcd8478cf-jg8sp"]
Oct 30 22:27:00.562792 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:00.562385    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-c1e2ab-openshift-default-bcd8478cf-jg8sp" podUID="ea11c8ac-4a65-463d-8956-739f3ae20611" containerName="istio-proxy" containerID="cri-o://63e08c843faa7a7f6333e45958d7c45f971dc9d933d75456abf77004208375ad" gracePeriod=30
Oct 30 22:27:01.609514 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:01.609479    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-14af5c-fk9jt"]
Oct 30 22:27:01.609995 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:01.609982    5061 cpu_manager.go:414] "containerMap: removing container" podUID="ba3aed70-33f3-4f0f-87cd-a8c19c347cd8" containerName="istio-proxy"
Oct 30 22:27:01.610042 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:01.609995    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba3aed70-33f3-4f0f-87cd-a8c19c347cd8" containerName="istio-proxy"
Oct 30 22:27:01.610042 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:01.610007    5061 cpu_manager.go:414] "containerMap: removing container" podUID="ba3aed70-33f3-4f0f-87cd-a8c19c347cd8" containerName="istio-proxy"
Oct 30 22:27:01.610042 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:01.610013    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba3aed70-33f3-4f0f-87cd-a8c19c347cd8" containerName="istio-proxy"
Oct 30 22:27:01.610042 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:01.610040    5061 cpu_manager.go:414] "containerMap: removing container" podUID="136b259c-03a5-4cc9-8702-fd78a552bb65" containerName="payload-pre-processing"
Oct 30 22:27:01.610181 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:01.610046    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="136b259c-03a5-4cc9-8702-fd78a552bb65" containerName="payload-pre-processing"
Oct 30 22:27:01.610181 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:01.610054    5061 cpu_manager.go:414] "containerMap: removing container" podUID="5e9f1392-e1ff-4491-af70-c0f0987ff9b9" containerName="revoke-keys"
Oct 30 22:27:01.610181 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:01.610059    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e9f1392-e1ff-4491-af70-c0f0987ff9b9" containerName="revoke-keys"
Oct 30 22:27:01.610181 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:01.610145    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="136b259c-03a5-4cc9-8702-fd78a552bb65" containerName="payload-pre-processing"
Oct 30 22:27:01.610181 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:01.610151    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e9f1392-e1ff-4491-af70-c0f0987ff9b9" containerName="revoke-keys"
Oct 30 22:27:01.610181 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:01.610162    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba3aed70-33f3-4f0f-87cd-a8c19c347cd8" containerName="istio-proxy"
Oct 30 22:27:01.610181 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:01.610167    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba3aed70-33f3-4f0f-87cd-a8c19c347cd8" containerName="istio-proxy"
Oct 30 22:27:01.613968 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:01.613945    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-14af5c-fk9jt"
Oct 30 22:27:01.663258 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:01.663229    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:27:01.667446 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:01.667419    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de890e38-beb5-4920-9525-66f7d57d2216-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-14af5c-fk9jt\" (UID: \"de890e38-beb5-4920-9525-66f7d57d2216\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-14af5c-fk9jt"
Oct 30 22:27:01.667603 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:01.667461    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvx8q\" (UniqueName: \"kubernetes.io/projected/de890e38-beb5-4920-9525-66f7d57d2216-kube-api-access-nvx8q\") pod \"maas-api-revoke-keys-e2e-worker-w6-14af5c-fk9jt\" (UID: \"de890e38-beb5-4920-9525-66f7d57d2216\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-14af5c-fk9jt"
Oct 30 22:27:01.667603 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:01.667581    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de890e38-beb5-4920-9525-66f7d57d2216-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-14af5c-fk9jt\" (UID: \"de890e38-beb5-4920-9525-66f7d57d2216\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-14af5c-fk9jt"
Oct 30 22:27:01.710079 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:01.710051    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gdmfg\""
Oct 30 22:27:01.713400 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:01.713379    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-14af5c-fk9jt"]
Oct 30 22:27:01.769188 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:01.769154    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de890e38-beb5-4920-9525-66f7d57d2216-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-14af5c-fk9jt\" (UID: \"de890e38-beb5-4920-9525-66f7d57d2216\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-14af5c-fk9jt"
Oct 30 22:27:01.769386 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:01.769195    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvx8q\" (UniqueName: \"kubernetes.io/projected/de890e38-beb5-4920-9525-66f7d57d2216-kube-api-access-nvx8q\") pod \"maas-api-revoke-keys-e2e-worker-w6-14af5c-fk9jt\" (UID: \"de890e38-beb5-4920-9525-66f7d57d2216\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-14af5c-fk9jt"
Oct 30 22:27:01.769386 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:01.769237    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de890e38-beb5-4920-9525-66f7d57d2216-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-14af5c-fk9jt\" (UID: \"de890e38-beb5-4920-9525-66f7d57d2216\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-14af5c-fk9jt"
Oct 30 22:27:01.769870 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:01.769850    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de890e38-beb5-4920-9525-66f7d57d2216-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-14af5c-fk9jt\" (UID: \"de890e38-beb5-4920-9525-66f7d57d2216\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-14af5c-fk9jt"
Oct 30 22:27:01.771599 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:01.771579    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de890e38-beb5-4920-9525-66f7d57d2216-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-14af5c-fk9jt\" (UID: \"de890e38-beb5-4920-9525-66f7d57d2216\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-14af5c-fk9jt"
Oct 30 22:27:01.823221 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:01.823190    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvx8q\" (UniqueName: \"kubernetes.io/projected/de890e38-beb5-4920-9525-66f7d57d2216-kube-api-access-nvx8q\") pod \"maas-api-revoke-keys-e2e-worker-w6-14af5c-fk9jt\" (UID: \"de890e38-beb5-4920-9525-66f7d57d2216\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-14af5c-fk9jt"
Oct 30 22:27:01.926975 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:01.926890    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-14af5c-fk9jt"
Oct 30 22:27:02.301343 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:02.301308    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-14af5c-fk9jt"]
Oct 30 22:27:02.301867 ip-10-0-130-60 kubenswrapper[5061]: W1030 22:27:02.301831    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde890e38_beb5_4920_9525_66f7d57d2216.slice/crio-32c098152626ce5b684722080855d2a669695070f820076a7a049b9bf77b1c31 WatchSource:0}: Error finding container 32c098152626ce5b684722080855d2a669695070f820076a7a049b9bf77b1c31: Status 404 returned error can't find the container with id 32c098152626ce5b684722080855d2a669695070f820076a7a049b9bf77b1c31
Oct 30 22:27:03.207823 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:03.207795    5061 generic.go:356] "Generic (PLEG): container finished" podID="de890e38-beb5-4920-9525-66f7d57d2216" containerID="3bababed699bab326bdbf355801c461dfd5f6fc3046bbb4a5b4cff5327a8f623" exitCode=0
Oct 30 22:27:03.208165 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:03.207872    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-14af5c-fk9jt" event={"ID":"de890e38-beb5-4920-9525-66f7d57d2216","Type":"ContainerDied","Data":"3bababed699bab326bdbf355801c461dfd5f6fc3046bbb4a5b4cff5327a8f623"}
Oct 30 22:27:03.208165 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:03.207924    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-14af5c-fk9jt" event={"ID":"de890e38-beb5-4920-9525-66f7d57d2216","Type":"ContainerStarted","Data":"32c098152626ce5b684722080855d2a669695070f820076a7a049b9bf77b1c31"}
Oct 30 22:27:04.357911 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:04.357885    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-14af5c-fk9jt"
Oct 30 22:27:04.399193 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:04.399167    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de890e38-beb5-4920-9525-66f7d57d2216-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de890e38-beb5-4920-9525-66f7d57d2216-maas-api-xks-ca\") pod \"de890e38-beb5-4920-9525-66f7d57d2216\" (UID: \"de890e38-beb5-4920-9525-66f7d57d2216\") "
Oct 30 22:27:04.399334 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:04.399229    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de890e38-beb5-4920-9525-66f7d57d2216-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de890e38-beb5-4920-9525-66f7d57d2216-maas-api-openshift-ca\") pod \"de890e38-beb5-4920-9525-66f7d57d2216\" (UID: \"de890e38-beb5-4920-9525-66f7d57d2216\") "
Oct 30 22:27:04.399334 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:04.399275    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de890e38-beb5-4920-9525-66f7d57d2216-kube-api-access-nvx8q\" (UniqueName: \"kubernetes.io/projected/de890e38-beb5-4920-9525-66f7d57d2216-kube-api-access-nvx8q\") pod \"de890e38-beb5-4920-9525-66f7d57d2216\" (UID: \"de890e38-beb5-4920-9525-66f7d57d2216\") "
Oct 30 22:27:04.399608 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:04.399585    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de890e38-beb5-4920-9525-66f7d57d2216-maas-api-openshift-ca" pod "de890e38-beb5-4920-9525-66f7d57d2216" (UID: "de890e38-beb5-4920-9525-66f7d57d2216"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:27:04.401262 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:04.401235    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de890e38-beb5-4920-9525-66f7d57d2216-maas-api-xks-ca" pod "de890e38-beb5-4920-9525-66f7d57d2216" (UID: "de890e38-beb5-4920-9525-66f7d57d2216"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:27:04.401342 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:04.401300    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de890e38-beb5-4920-9525-66f7d57d2216-kube-api-access-nvx8q" pod "de890e38-beb5-4920-9525-66f7d57d2216" (UID: "de890e38-beb5-4920-9525-66f7d57d2216"). InnerVolumeSpecName "kube-api-access-nvx8q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:27:04.501156 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:04.501094    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de890e38-beb5-4920-9525-66f7d57d2216-maas-api-xks-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:04.501156 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:04.501121    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de890e38-beb5-4920-9525-66f7d57d2216-maas-api-openshift-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:04.501156 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:04.501131    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvx8q\" (UniqueName: \"kubernetes.io/projected/de890e38-beb5-4920-9525-66f7d57d2216-kube-api-access-nvx8q\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:05.221836 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:05.221805    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-14af5c-fk9jt"
Oct 30 22:27:05.222024 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:05.221846    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-14af5c-fk9jt" event={"ID":"de890e38-beb5-4920-9525-66f7d57d2216","Type":"ContainerDied","Data":"32c098152626ce5b684722080855d2a669695070f820076a7a049b9bf77b1c31"}
Oct 30 22:27:05.222024 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:05.221893    5061 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="32c098152626ce5b684722080855d2a669695070f820076a7a049b9bf77b1c31"
Oct 30 22:27:05.835337 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:05.835282    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c1e2ab-openshift-default-bcd8478cf-jg8sp"
Oct 30 22:27:05.916736 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:05.916713    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ea11c8ac-4a65-463d-8956-739f3ae20611-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea11c8ac-4a65-463d-8956-739f3ae20611-istio-podinfo\") pod \"ea11c8ac-4a65-463d-8956-739f3ae20611\" (UID: \"ea11c8ac-4a65-463d-8956-739f3ae20611\") "
Oct 30 22:27:05.916883 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:05.916746    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea11c8ac-4a65-463d-8956-739f3ae20611-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea11c8ac-4a65-463d-8956-739f3ae20611-workload-socket\") pod \"ea11c8ac-4a65-463d-8956-739f3ae20611\" (UID: \"ea11c8ac-4a65-463d-8956-739f3ae20611\") "
Oct 30 22:27:05.916883 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:05.916797    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea11c8ac-4a65-463d-8956-739f3ae20611-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea11c8ac-4a65-463d-8956-739f3ae20611-istio-data\") pod \"ea11c8ac-4a65-463d-8956-739f3ae20611\" (UID: \"ea11c8ac-4a65-463d-8956-739f3ae20611\") "
Oct 30 22:27:05.916997 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:05.916971    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea11c8ac-4a65-463d-8956-739f3ae20611-istio-token\" (UniqueName: \"kubernetes.io/projected/ea11c8ac-4a65-463d-8956-739f3ae20611-istio-token\") pod \"ea11c8ac-4a65-463d-8956-739f3ae20611\" (UID: \"ea11c8ac-4a65-463d-8956-739f3ae20611\") "
Oct 30 22:27:05.917065 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:05.917018    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea11c8ac-4a65-463d-8956-739f3ae20611-istio-data" pod "ea11c8ac-4a65-463d-8956-739f3ae20611" (UID: "ea11c8ac-4a65-463d-8956-739f3ae20611"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:27:05.917065 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:05.917034    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea11c8ac-4a65-463d-8956-739f3ae20611-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea11c8ac-4a65-463d-8956-739f3ae20611-credential-socket\") pod \"ea11c8ac-4a65-463d-8956-739f3ae20611\" (UID: \"ea11c8ac-4a65-463d-8956-739f3ae20611\") "
Oct 30 22:27:05.917065 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:05.917051    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea11c8ac-4a65-463d-8956-739f3ae20611-workload-socket" pod "ea11c8ac-4a65-463d-8956-739f3ae20611" (UID: "ea11c8ac-4a65-463d-8956-739f3ae20611"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:27:05.917229 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:05.917092    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea11c8ac-4a65-463d-8956-739f3ae20611-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea11c8ac-4a65-463d-8956-739f3ae20611-istiod-ca-cert\") pod \"ea11c8ac-4a65-463d-8956-739f3ae20611\" (UID: \"ea11c8ac-4a65-463d-8956-739f3ae20611\") "
Oct 30 22:27:05.917229 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:05.917145    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea11c8ac-4a65-463d-8956-739f3ae20611-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea11c8ac-4a65-463d-8956-739f3ae20611-istio-envoy\") pod \"ea11c8ac-4a65-463d-8956-739f3ae20611\" (UID: \"ea11c8ac-4a65-463d-8956-739f3ae20611\") "
Oct 30 22:27:05.917229 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:05.917199    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea11c8ac-4a65-463d-8956-739f3ae20611-kube-api-access-s4nwz\" (UniqueName: \"kubernetes.io/projected/ea11c8ac-4a65-463d-8956-739f3ae20611-kube-api-access-s4nwz\") pod \"ea11c8ac-4a65-463d-8956-739f3ae20611\" (UID: \"ea11c8ac-4a65-463d-8956-739f3ae20611\") "
Oct 30 22:27:05.917229 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:05.917217    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea11c8ac-4a65-463d-8956-739f3ae20611-credential-socket" pod "ea11c8ac-4a65-463d-8956-739f3ae20611" (UID: "ea11c8ac-4a65-463d-8956-739f3ae20611"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:27:05.917454 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:05.917259    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea11c8ac-4a65-463d-8956-739f3ae20611-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea11c8ac-4a65-463d-8956-739f3ae20611-workload-certs\") pod \"ea11c8ac-4a65-463d-8956-739f3ae20611\" (UID: \"ea11c8ac-4a65-463d-8956-739f3ae20611\") "
Oct 30 22:27:05.917509 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:05.917471    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea11c8ac-4a65-463d-8956-739f3ae20611-istiod-ca-cert" pod "ea11c8ac-4a65-463d-8956-739f3ae20611" (UID: "ea11c8ac-4a65-463d-8956-739f3ae20611"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:27:05.917700 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:05.917677    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea11c8ac-4a65-463d-8956-739f3ae20611-workload-certs" pod "ea11c8ac-4a65-463d-8956-739f3ae20611" (UID: "ea11c8ac-4a65-463d-8956-739f3ae20611"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:27:05.917914 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:05.917892    5061 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea11c8ac-4a65-463d-8956-739f3ae20611-workload-certs\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:05.917981 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:05.917922    5061 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea11c8ac-4a65-463d-8956-739f3ae20611-workload-socket\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:05.917981 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:05.917939    5061 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea11c8ac-4a65-463d-8956-739f3ae20611-istio-data\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:05.917981 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:05.917953    5061 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea11c8ac-4a65-463d-8956-739f3ae20611-credential-socket\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:05.917981 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:05.917968    5061 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea11c8ac-4a65-463d-8956-739f3ae20611-istiod-ca-cert\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:05.919217 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:05.919193    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea11c8ac-4a65-463d-8956-739f3ae20611-istio-envoy" pod "ea11c8ac-4a65-463d-8956-739f3ae20611" (UID: "ea11c8ac-4a65-463d-8956-739f3ae20611"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:27:05.919344 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:05.919240    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea11c8ac-4a65-463d-8956-739f3ae20611-istio-token" pod "ea11c8ac-4a65-463d-8956-739f3ae20611" (UID: "ea11c8ac-4a65-463d-8956-739f3ae20611"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:27:05.919490 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:05.919467    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ea11c8ac-4a65-463d-8956-739f3ae20611-istio-podinfo" pod "ea11c8ac-4a65-463d-8956-739f3ae20611" (UID: "ea11c8ac-4a65-463d-8956-739f3ae20611"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:27:05.919639 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:05.919594    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea11c8ac-4a65-463d-8956-739f3ae20611-kube-api-access-s4nwz" pod "ea11c8ac-4a65-463d-8956-739f3ae20611" (UID: "ea11c8ac-4a65-463d-8956-739f3ae20611"). InnerVolumeSpecName "kube-api-access-s4nwz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:27:06.019350 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:06.019316    5061 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea11c8ac-4a65-463d-8956-739f3ae20611-istio-podinfo\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:06.019350 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:06.019344    5061 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea11c8ac-4a65-463d-8956-739f3ae20611-istio-token\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:06.019350 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:06.019354    5061 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea11c8ac-4a65-463d-8956-739f3ae20611-istio-envoy\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:06.019704 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:06.019363    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4nwz\" (UniqueName: \"kubernetes.io/projected/ea11c8ac-4a65-463d-8956-739f3ae20611-kube-api-access-s4nwz\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:06.046984 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:06.046948    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-14af5c/distinct-2-w6-14af5c-kserve-58d554bdb6-r5r6j"]
Oct 30 22:27:06.055491 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:06.054287    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-14af5c/distinct-w6-14af5c-kserve-7c7cb9d69c-hljl9"]
Oct 30 22:27:06.067479 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:06.067444    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-14af5c/premium-w6-14af5c-kserve-6549647f4-f4gwg"]
Oct 30 22:27:06.113410 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:06.113341    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-14af5c/simulator-w6-14af5c-kserve-85b8bd96d7-wlcb9"]
Oct 30 22:27:06.233047 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:06.233012    5061 generic.go:356] "Generic (PLEG): container finished" podID="ea11c8ac-4a65-463d-8956-739f3ae20611" containerID="63e08c843faa7a7f6333e45958d7c45f971dc9d933d75456abf77004208375ad" exitCode=0
Oct 30 22:27:06.233245 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:06.233064    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c1e2ab-openshift-default-bcd8478cf-jg8sp"
Oct 30 22:27:06.233245 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:06.233079    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c1e2ab-openshift-default-bcd8478cf-jg8sp" event={"ID":"ea11c8ac-4a65-463d-8956-739f3ae20611","Type":"ContainerDied","Data":"63e08c843faa7a7f6333e45958d7c45f971dc9d933d75456abf77004208375ad"}
Oct 30 22:27:06.233245 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:06.233128    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c1e2ab-openshift-default-bcd8478cf-jg8sp" event={"ID":"ea11c8ac-4a65-463d-8956-739f3ae20611","Type":"ContainerDied","Data":"cc593246e8c0f6f2d66b58031fa1fe4561c2ba224a8a7eb0b4e523f864dc3c43"}
Oct 30 22:27:06.233245 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:06.233161    5061 scope.go:122] "RemoveContainer" containerID="63e08c843faa7a7f6333e45958d7c45f971dc9d933d75456abf77004208375ad"
Oct 30 22:27:06.244485 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:06.244466    5061 scope.go:122] "RemoveContainer" containerID="63e08c843faa7a7f6333e45958d7c45f971dc9d933d75456abf77004208375ad"
Oct 30 22:27:06.244796 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:27:06.244777    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63e08c843faa7a7f6333e45958d7c45f971dc9d933d75456abf77004208375ad\": container with ID starting with 63e08c843faa7a7f6333e45958d7c45f971dc9d933d75456abf77004208375ad not found: ID does not exist" containerID="63e08c843faa7a7f6333e45958d7c45f971dc9d933d75456abf77004208375ad"
Oct 30 22:27:06.244885 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:06.244804    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63e08c843faa7a7f6333e45958d7c45f971dc9d933d75456abf77004208375ad"} err="failed to get container status \"63e08c843faa7a7f6333e45958d7c45f971dc9d933d75456abf77004208375ad\": rpc error: code = NotFound desc = could not find container \"63e08c843faa7a7f6333e45958d7c45f971dc9d933d75456abf77004208375ad\": container with ID starting with 63e08c843faa7a7f6333e45958d7c45f971dc9d933d75456abf77004208375ad not found: ID does not exist"
Oct 30 22:27:06.553859 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:06.553830    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-c1e2ab-openshift-default-bcd8478cf-jg8sp"]
Oct 30 22:27:06.559714 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:06.559689    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-c1e2ab-openshift-default-bcd8478cf-jg8sp"]
Oct 30 22:27:08.528113 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:08.528076    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea11c8ac-4a65-463d-8956-739f3ae20611" path="/var/lib/kubelet/pods/ea11c8ac-4a65-463d-8956-739f3ae20611/volumes"
Oct 30 22:27:13.046162 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:13.046122    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-14af5c-85d68c5546-r4slw"]
Oct 30 22:27:18.046696 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:18.046656    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w6-14af5c-85d68c5546-r4slw" podUID="6977b446-b5f9-4f27-8d41-a4942379c9b2" containerName="payload-processing" containerID="cri-o://86d8a666c1779dda82bfa30c16d519a8d51af7e4b1517f6f22318facc31637d1" gracePeriod=25
Oct 30 22:27:20.700663 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:20.700623    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-14af5c-85d68c5546-r4slw" podUID="6977b446-b5f9-4f27-8d41-a4942379c9b2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:27:21.070121 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:21.070080    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-14af5c/distinct-2-w6-14af5c-kserve-58d554bdb6-r5r6j" podUID="990f4e94-708e-45f3-b2ad-01cd50b3257b" containerName="main" containerID="cri-o://5e7910050fbfcac88f9f0834d65fc867bbb60d23395ae586e81639e7953f80cd" gracePeriod=45
Oct 30 22:27:21.090537 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:21.090490    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-14af5c/premium-w6-14af5c-kserve-6549647f4-f4gwg" podUID="9b42ee01-c60e-462c-a0e0-b04a52014db6" containerName="main" containerID="cri-o://b1b2212c3c7180a7832363d0cc2347107f1744364b4e0b1561dd4a8c2fd722cf" gracePeriod=45
Oct 30 22:27:21.133182 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:21.133138    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-14af5c/simulator-w6-14af5c-kserve-85b8bd96d7-wlcb9" podUID="39766c7d-5f1d-42db-8920-c0321cd12c99" containerName="main" containerID="cri-o://4742be0ecaf01e860e10e4751e0a2620e7d66623622e6cc0756908c05a65b260" gracePeriod=45
Oct 30 22:27:21.254471 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:21.253871    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-14af5c/distinct-w6-14af5c-kserve-7c7cb9d69c-hljl9" podUID="1f8d9486-fb2f-491a-a8d7-c23c21bd116a" containerName="main" containerID="cri-o://457c372be65206f6c139509e0f4926caeefaf8a0848f0b8831e44587c273745f" gracePeriod=45
Oct 30 22:27:21.335891 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:21.335848    5061 generic.go:356] "Generic (PLEG): container finished" podID="39766c7d-5f1d-42db-8920-c0321cd12c99" containerID="4742be0ecaf01e860e10e4751e0a2620e7d66623622e6cc0756908c05a65b260" exitCode=0
Oct 30 22:27:21.336085 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:21.335989    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-14af5c/simulator-w6-14af5c-kserve-85b8bd96d7-wlcb9" event={"ID":"39766c7d-5f1d-42db-8920-c0321cd12c99","Type":"ContainerDied","Data":"4742be0ecaf01e860e10e4751e0a2620e7d66623622e6cc0756908c05a65b260"}
Oct 30 22:27:21.338153 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:21.338122    5061 generic.go:356] "Generic (PLEG): container finished" podID="990f4e94-708e-45f3-b2ad-01cd50b3257b" containerID="5e7910050fbfcac88f9f0834d65fc867bbb60d23395ae586e81639e7953f80cd" exitCode=0
Oct 30 22:27:21.338301 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:21.338252    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-14af5c/distinct-2-w6-14af5c-kserve-58d554bdb6-r5r6j" event={"ID":"990f4e94-708e-45f3-b2ad-01cd50b3257b","Type":"ContainerDied","Data":"5e7910050fbfcac88f9f0834d65fc867bbb60d23395ae586e81639e7953f80cd"}
Oct 30 22:27:21.340087 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:21.339955    5061 generic.go:356] "Generic (PLEG): container finished" podID="9b42ee01-c60e-462c-a0e0-b04a52014db6" containerID="b1b2212c3c7180a7832363d0cc2347107f1744364b4e0b1561dd4a8c2fd722cf" exitCode=0
Oct 30 22:27:21.340087 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:21.340010    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-14af5c/premium-w6-14af5c-kserve-6549647f4-f4gwg" event={"ID":"9b42ee01-c60e-462c-a0e0-b04a52014db6","Type":"ContainerDied","Data":"b1b2212c3c7180a7832363d0cc2347107f1744364b4e0b1561dd4a8c2fd722cf"}
Oct 30 22:27:21.420804 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:21.420757    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-14af5c/distinct-2-w6-14af5c-kserve-58d554bdb6-r5r6j"
Oct 30 22:27:21.441073 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:21.441043    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-14af5c/premium-w6-14af5c-kserve-6549647f4-f4gwg"
Oct 30 22:27:21.451989 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:21.451969    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-14af5c/simulator-w6-14af5c-kserve-85b8bd96d7-wlcb9"
Oct 30 22:27:21.549856 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:21.549764    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-14af5c/distinct-w6-14af5c-kserve-7c7cb9d69c-hljl9"
Oct 30 22:27:21.597040 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:21.596950    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39766c7d-5f1d-42db-8920-c0321cd12c99-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39766c7d-5f1d-42db-8920-c0321cd12c99-model-cache\") pod \"39766c7d-5f1d-42db-8920-c0321cd12c99\" (UID: \"39766c7d-5f1d-42db-8920-c0321cd12c99\") "
Oct 30 22:27:21.597040 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:21.597019    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/990f4e94-708e-45f3-b2ad-01cd50b3257b-kube-api-access-hqqsw\" (UniqueName: \"kubernetes.io/projected/990f4e94-708e-45f3-b2ad-01cd50b3257b-kube-api-access-hqqsw\") pod \"990f4e94-708e-45f3-b2ad-01cd50b3257b\" (UID: \"990f4e94-708e-45f3-b2ad-01cd50b3257b\") "
Oct 30 22:27:21.597274 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:21.597085    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/990f4e94-708e-45f3-b2ad-01cd50b3257b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/990f4e94-708e-45f3-b2ad-01cd50b3257b-dshm\") pod \"990f4e94-708e-45f3-b2ad-01cd50b3257b\" (UID: \"990f4e94-708e-45f3-b2ad-01cd50b3257b\") "
Oct 30 22:27:21.597274 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:21.597126    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39766c7d-5f1d-42db-8920-c0321cd12c99-tls-certs\" (UniqueName: \"kubernetes.io/secret/39766c7d-5f1d-42db-8920-c0321cd12c99-tls-certs\") pod \"39766c7d-5f1d-42db-8920-c0321cd12c99\" (UID: \"39766c7d-5f1d-42db-8920-c0321cd12c99\") "
Oct 30 22:27:21.597274 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:21.597159    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/990f4e94-708e-45f3-b2ad-01cd50b3257b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/990f4e94-708e-45f3-b2ad-01cd50b3257b-model-cache\") pod \"990f4e94-708e-45f3-b2ad-01cd50b3257b\" (UID: \"990f4e94-708e-45f3-b2ad-01cd50b3257b\") "
Oct 30 22:27:21.597274 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:21.597194    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b42ee01-c60e-462c-a0e0-b04a52014db6-tls-certs\" (UniqueName: \"kubernetes.io/secret/9b42ee01-c60e-462c-a0e0-b04a52014db6-tls-certs\") pod \"9b42ee01-c60e-462c-a0e0-b04a52014db6\" (UID: \"9b42ee01-c60e-462c-a0e0-b04a52014db6\") "
Oct 30 22:27:21.597274 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:21.597233    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b42ee01-c60e-462c-a0e0-b04a52014db6-home\" (UniqueName: \"kubernetes.io/empty-dir/9b42ee01-c60e-462c-a0e0-b04a52014db6-home\") pod \"9b42ee01-c60e-462c-a0e0-b04a52014db6\" (UID: \"9b42ee01-c60e-462c-a0e0-b04a52014db6\") "
Oct 30 22:27:21.597274 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:21.597229    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39766c7d-5f1d-42db-8920-c0321cd12c99-model-cache" pod "39766c7d-5f1d-42db-8920-c0321cd12c99" (UID: "39766c7d-5f1d-42db-8920-c0321cd12c99"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:27:21.597651 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:21.597284    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b42ee01-c60e-462c-a0e0-b04a52014db6-kube-api-access-2p6cm\" (UniqueName: \"kubernetes.io/projected/9b42ee01-c60e-462c-a0e0-b04a52014db6-kube-api-access-2p6cm\") pod \"9b42ee01-c60e-462c-a0e0-b04a52014db6\" (UID: \"9b42ee01-c60e-462c-a0e0-b04a52014db6\") "
Oct 30 22:27:21.597651 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:21.597366    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b42ee01-c60e-462c-a0e0-b04a52014db6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b42ee01-c60e-462c-a0e0-b04a52014db6-model-cache\") pod \"9b42ee01-c60e-462c-a0e0-b04a52014db6\" (UID: \"9b42ee01-c60e-462c-a0e0-b04a52014db6\") "
Oct 30 22:27:21.597651 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:21.597467    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39766c7d-5f1d-42db-8920-c0321cd12c99-dshm\" (UniqueName: \"kubernetes.io/empty-dir/39766c7d-5f1d-42db-8920-c0321cd12c99-dshm\") pod \"39766c7d-5f1d-42db-8920-c0321cd12c99\" (UID: \"39766c7d-5f1d-42db-8920-c0321cd12c99\") "
Oct 30 22:27:21.597651 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:21.597473    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/990f4e94-708e-45f3-b2ad-01cd50b3257b-model-cache" pod "990f4e94-708e-45f3-b2ad-01cd50b3257b" (UID: "990f4e94-708e-45f3-b2ad-01cd50b3257b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:27:21.597651 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:21.597502    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/990f4e94-708e-45f3-b2ad-01cd50b3257b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/990f4e94-708e-45f3-b2ad-01cd50b3257b-tmp-dir\") pod \"990f4e94-708e-45f3-b2ad-01cd50b3257b\" (UID: \"990f4e94-708e-45f3-b2ad-01cd50b3257b\") "
Oct 30 22:27:21.597651 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:21.597557    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/990f4e94-708e-45f3-b2ad-01cd50b3257b-tls-certs\" (UniqueName: \"kubernetes.io/secret/990f4e94-708e-45f3-b2ad-01cd50b3257b-tls-certs\") pod \"990f4e94-708e-45f3-b2ad-01cd50b3257b\" (UID: \"990f4e94-708e-45f3-b2ad-01cd50b3257b\") "
Oct 30 22:27:21.597651 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:21.597591    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b42ee01-c60e-462c-a0e0-b04a52014db6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b42ee01-c60e-462c-a0e0-b04a52014db6-dshm\") pod \"9b42ee01-c60e-462c-a0e0-b04a52014db6\" (UID: \"9b42ee01-c60e-462c-a0e0-b04a52014db6\") "
Oct 30 22:27:21.598065 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:21.597638    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b42ee01-c60e-462c-a0e0-b04a52014db6-home" pod "9b42ee01-c60e-462c-a0e0-b04a52014db6" (UID: "9b42ee01-c60e-462c-a0e0-b04a52014db6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:27:21.598065 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:21.597665    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/990f4e94-708e-45f3-b2ad-01cd50b3257b-home\" (UniqueName: \"kubernetes.io/empty-dir/990f4e94-708e-45f3-b2ad-01cd50b3257b-home\") pod \"990f4e94-708e-45f3-b2ad-01cd50b3257b\" (UID: \"990f4e94-708e-45f3-b2ad-01cd50b3257b\") "
Oct 30 22:27:21.598065 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:21.597708    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b42ee01-c60e-462c-a0e0-b04a52014db6-model-cache" pod "9b42ee01-c60e-462c-a0e0-b04a52014db6" (UID: "9b42ee01-c60e-462c-a0e0-b04a52014db6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:27:21.598065 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:21.597731    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39766c7d-5f1d-42db-8920-c0321cd12c99-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39766c7d-5f1d-42db-8920-c0321cd12c99-tmp-dir\") pod \"39766c7d-5f1d-42db-8920-c0321cd12c99\" (UID: \"39766c7d-5f1d-42db-8920-c0321cd12c99\") "
Oct 30 22:27:21.598065 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:21.597780    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39766c7d-5f1d-42db-8920-c0321cd12c99-home\" (UniqueName: \"kubernetes.io/empty-dir/39766c7d-5f1d-42db-8920-c0321cd12c99-home\") pod \"39766c7d-5f1d-42db-8920-c0321cd12c99\" (UID: \"39766c7d-5f1d-42db-8920-c0321cd12c99\") "
Oct 30 22:27:21.598065 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:21.597829    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39766c7d-5f1d-42db-8920-c0321cd12c99-kube-api-access-5rcgj\" (UniqueName: \"kubernetes.io/projected/39766c7d-5f1d-42db-8920-c0321cd12c99-kube-api-access-5rcgj\") pod \"39766c7d-5f1d-42db-8920-c0321cd12c99\" (UID: \"39766c7d-5f1d-42db-8920-c0321cd12c99\") "
Oct 30 22:27:21.598065 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:21.597878    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b42ee01-c60e-462c-a0e0-b04a52014db6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b42ee01-c60e-462c-a0e0-b04a52014db6-tmp-dir\") pod \"9b42ee01-c60e-462c-a0e0-b04a52014db6\" (UID: \"9b42ee01-c60e-462c-a0e0-b04a52014db6\") "
Oct 30 22:27:21.598065 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:21.598024    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/990f4e94-708e-45f3-b2ad-01cd50b3257b-tmp-dir" pod "990f4e94-708e-45f3-b2ad-01cd50b3257b" (UID: "990f4e94-708e-45f3-b2ad-01cd50b3257b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:27:21.598641 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:21.598262    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b42ee01-c60e-462c-a0e0-b04a52014db6-tmp-dir" pod "9b42ee01-c60e-462c-a0e0-b04a52014db6" (UID: "9b42ee01-c60e-462c-a0e0-b04a52014db6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:27:21.598641 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:21.598600    5061 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/990f4e94-708e-45f3-b2ad-01cd50b3257b-model-cache\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:21.598775 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:21.598642    5061 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b42ee01-c60e-462c-a0e0-b04a52014db6-home\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:21.598775 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:21.598657    5061 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b42ee01-c60e-462c-a0e0-b04a52014db6-model-cache\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:21.598775 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:21.598673    5061 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/990f4e94-708e-45f3-b2ad-01cd50b3257b-tmp-dir\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:21.598775 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:21.598705    5061 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b42ee01-c60e-462c-a0e0-b04a52014db6-tmp-dir\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:21.598775 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:21.598720    5061 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39766c7d-5f1d-42db-8920-c0321cd12c99-model-cache\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:21.599059 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:21.598981    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39766c7d-5f1d-42db-8920-c0321cd12c99-tmp-dir" pod "39766c7d-5f1d-42db-8920-c0321cd12c99" (UID: "39766c7d-5f1d-42db-8920-c0321cd12c99"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:27:21.599811 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:21.599787    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/990f4e94-708e-45f3-b2ad-01cd50b3257b-home" pod "990f4e94-708e-45f3-b2ad-01cd50b3257b" (UID: "990f4e94-708e-45f3-b2ad-01cd50b3257b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:27:21.599983 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:21.599962    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/990f4e94-708e-45f3-b2ad-01cd50b3257b-dshm" pod "990f4e94-708e-45f3-b2ad-01cd50b3257b" (UID: "990f4e94-708e-45f3-b2ad-01cd50b3257b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:27:21.600066 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:21.600009    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39766c7d-5f1d-42db-8920-c0321cd12c99-home" pod "39766c7d-5f1d-42db-8920-c0321cd12c99" (UID: "39766c7d-5f1d-42db-8920-c0321cd12c99"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:27:21.601094 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:21.601071    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39766c7d-5f1d-42db-8920-c0321cd12c99-dshm" pod "39766c7d-5f1d-42db-8920-c0321cd12c99" (UID: "39766c7d-5f1d-42db-8920-c0321cd12c99"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:27:21.601306 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:21.601100    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/990f4e94-708e-45f3-b2ad-01cd50b3257b-tls-certs" pod "990f4e94-708e-45f3-b2ad-01cd50b3257b" (UID: "990f4e94-708e-45f3-b2ad-01cd50b3257b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:27:21.601306 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:21.601179    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b42ee01-c60e-462c-a0e0-b04a52014db6-kube-api-access-2p6cm" pod "9b42ee01-c60e-462c-a0e0-b04a52014db6" (UID: "9b42ee01-c60e-462c-a0e0-b04a52014db6"). InnerVolumeSpecName "kube-api-access-2p6cm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:27:21.601306 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:21.601283    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/990f4e94-708e-45f3-b2ad-01cd50b3257b-kube-api-access-hqqsw" pod "990f4e94-708e-45f3-b2ad-01cd50b3257b" (UID: "990f4e94-708e-45f3-b2ad-01cd50b3257b"). InnerVolumeSpecName "kube-api-access-hqqsw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:27:21.601502 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:21.601485    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39766c7d-5f1d-42db-8920-c0321cd12c99-tls-certs" pod "39766c7d-5f1d-42db-8920-c0321cd12c99" (UID: "39766c7d-5f1d-42db-8920-c0321cd12c99"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:27:21.601617 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:21.601592    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b42ee01-c60e-462c-a0e0-b04a52014db6-dshm" pod "9b42ee01-c60e-462c-a0e0-b04a52014db6" (UID: "9b42ee01-c60e-462c-a0e0-b04a52014db6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:27:21.601985 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:21.601952    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b42ee01-c60e-462c-a0e0-b04a52014db6-tls-certs" pod "9b42ee01-c60e-462c-a0e0-b04a52014db6" (UID: "9b42ee01-c60e-462c-a0e0-b04a52014db6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:27:21.602656 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:21.602639    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39766c7d-5f1d-42db-8920-c0321cd12c99-kube-api-access-5rcgj" pod "39766c7d-5f1d-42db-8920-c0321cd12c99" (UID: "39766c7d-5f1d-42db-8920-c0321cd12c99"). InnerVolumeSpecName "kube-api-access-5rcgj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:27:21.700018 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:21.699986    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f8d9486-fb2f-491a-a8d7-c23c21bd116a-kube-api-access-p68cd\" (UniqueName: \"kubernetes.io/projected/1f8d9486-fb2f-491a-a8d7-c23c21bd116a-kube-api-access-p68cd\") pod \"1f8d9486-fb2f-491a-a8d7-c23c21bd116a\" (UID: \"1f8d9486-fb2f-491a-a8d7-c23c21bd116a\") "
Oct 30 22:27:21.700216 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:21.700027    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f8d9486-fb2f-491a-a8d7-c23c21bd116a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f8d9486-fb2f-491a-a8d7-c23c21bd116a-dshm\") pod \"1f8d9486-fb2f-491a-a8d7-c23c21bd116a\" (UID: \"1f8d9486-fb2f-491a-a8d7-c23c21bd116a\") "
Oct 30 22:27:21.700216 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:21.700100    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f8d9486-fb2f-491a-a8d7-c23c21bd116a-home\" (UniqueName: \"kubernetes.io/empty-dir/1f8d9486-fb2f-491a-a8d7-c23c21bd116a-home\") pod \"1f8d9486-fb2f-491a-a8d7-c23c21bd116a\" (UID: \"1f8d9486-fb2f-491a-a8d7-c23c21bd116a\") "
Oct 30 22:27:21.700216 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:21.700132    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f8d9486-fb2f-491a-a8d7-c23c21bd116a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f8d9486-fb2f-491a-a8d7-c23c21bd116a-tmp-dir\") pod \"1f8d9486-fb2f-491a-a8d7-c23c21bd116a\" (UID: \"1f8d9486-fb2f-491a-a8d7-c23c21bd116a\") "
Oct 30 22:27:21.700216 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:21.700199    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f8d9486-fb2f-491a-a8d7-c23c21bd116a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f8d9486-fb2f-491a-a8d7-c23c21bd116a-model-cache\") pod \"1f8d9486-fb2f-491a-a8d7-c23c21bd116a\" (UID: \"1f8d9486-fb2f-491a-a8d7-c23c21bd116a\") "
Oct 30 22:27:21.700477 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:21.700273    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f8d9486-fb2f-491a-a8d7-c23c21bd116a-tls-certs\" (UniqueName: \"kubernetes.io/secret/1f8d9486-fb2f-491a-a8d7-c23c21bd116a-tls-certs\") pod \"1f8d9486-fb2f-491a-a8d7-c23c21bd116a\" (UID: \"1f8d9486-fb2f-491a-a8d7-c23c21bd116a\") "
Oct 30 22:27:21.700477 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:21.700451    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f8d9486-fb2f-491a-a8d7-c23c21bd116a-tmp-dir" pod "1f8d9486-fb2f-491a-a8d7-c23c21bd116a" (UID: "1f8d9486-fb2f-491a-a8d7-c23c21bd116a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:27:21.700621 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:21.700569    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f8d9486-fb2f-491a-a8d7-c23c21bd116a-model-cache" pod "1f8d9486-fb2f-491a-a8d7-c23c21bd116a" (UID: "1f8d9486-fb2f-491a-a8d7-c23c21bd116a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:27:21.700859 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:21.700831    5061 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39766c7d-5f1d-42db-8920-c0321cd12c99-dshm\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:21.700859 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:21.700860    5061 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/990f4e94-708e-45f3-b2ad-01cd50b3257b-tls-certs\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:21.701369 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:21.700877    5061 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b42ee01-c60e-462c-a0e0-b04a52014db6-dshm\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:21.701369 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:21.700889    5061 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/990f4e94-708e-45f3-b2ad-01cd50b3257b-home\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:21.701369 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:21.700904    5061 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39766c7d-5f1d-42db-8920-c0321cd12c99-tmp-dir\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:21.701369 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:21.700918    5061 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39766c7d-5f1d-42db-8920-c0321cd12c99-home\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:21.701369 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:21.700919    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f8d9486-fb2f-491a-a8d7-c23c21bd116a-home" pod "1f8d9486-fb2f-491a-a8d7-c23c21bd116a" (UID: "1f8d9486-fb2f-491a-a8d7-c23c21bd116a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:27:21.701369 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:21.700933    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rcgj\" (UniqueName: \"kubernetes.io/projected/39766c7d-5f1d-42db-8920-c0321cd12c99-kube-api-access-5rcgj\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:21.701369 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:21.700971    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqqsw\" (UniqueName: \"kubernetes.io/projected/990f4e94-708e-45f3-b2ad-01cd50b3257b-kube-api-access-hqqsw\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:21.701369 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:21.700989    5061 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/990f4e94-708e-45f3-b2ad-01cd50b3257b-dshm\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:21.701369 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:21.701005    5061 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39766c7d-5f1d-42db-8920-c0321cd12c99-tls-certs\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:21.701369 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:21.701018    5061 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b42ee01-c60e-462c-a0e0-b04a52014db6-tls-certs\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:21.701369 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:21.701031    5061 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f8d9486-fb2f-491a-a8d7-c23c21bd116a-tmp-dir\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:21.701369 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:21.701044    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2p6cm\" (UniqueName: \"kubernetes.io/projected/9b42ee01-c60e-462c-a0e0-b04a52014db6-kube-api-access-2p6cm\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:21.701369 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:21.701058    5061 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f8d9486-fb2f-491a-a8d7-c23c21bd116a-model-cache\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:21.702246 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:21.702225    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f8d9486-fb2f-491a-a8d7-c23c21bd116a-kube-api-access-p68cd" pod "1f8d9486-fb2f-491a-a8d7-c23c21bd116a" (UID: "1f8d9486-fb2f-491a-a8d7-c23c21bd116a"). InnerVolumeSpecName "kube-api-access-p68cd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:27:21.702246 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:21.702238    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f8d9486-fb2f-491a-a8d7-c23c21bd116a-dshm" pod "1f8d9486-fb2f-491a-a8d7-c23c21bd116a" (UID: "1f8d9486-fb2f-491a-a8d7-c23c21bd116a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:27:21.702684 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:21.702660    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f8d9486-fb2f-491a-a8d7-c23c21bd116a-tls-certs" pod "1f8d9486-fb2f-491a-a8d7-c23c21bd116a" (UID: "1f8d9486-fb2f-491a-a8d7-c23c21bd116a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:27:21.802008 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:21.801972    5061 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f8d9486-fb2f-491a-a8d7-c23c21bd116a-tls-certs\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:21.802008 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:21.802008    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p68cd\" (UniqueName: \"kubernetes.io/projected/1f8d9486-fb2f-491a-a8d7-c23c21bd116a-kube-api-access-p68cd\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:21.802255 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:21.802027    5061 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f8d9486-fb2f-491a-a8d7-c23c21bd116a-dshm\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:21.802255 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:21.802041    5061 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f8d9486-fb2f-491a-a8d7-c23c21bd116a-home\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:22.348322 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:22.348234    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-14af5c/distinct-2-w6-14af5c-kserve-58d554bdb6-r5r6j"
Oct 30 22:27:22.348322 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:22.348267    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-14af5c/distinct-2-w6-14af5c-kserve-58d554bdb6-r5r6j" event={"ID":"990f4e94-708e-45f3-b2ad-01cd50b3257b","Type":"ContainerDied","Data":"b8ff36995f8badaa5e5c043ec4da00e517b557d03abdccc5954f223e69b370d8"}
Oct 30 22:27:22.348583 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:22.348352    5061 scope.go:122] "RemoveContainer" containerID="5e7910050fbfcac88f9f0834d65fc867bbb60d23395ae586e81639e7953f80cd"
Oct 30 22:27:22.349875 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:22.349799    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-14af5c/premium-w6-14af5c-kserve-6549647f4-f4gwg"
Oct 30 22:27:22.349875 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:22.349856    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-14af5c/premium-w6-14af5c-kserve-6549647f4-f4gwg" event={"ID":"9b42ee01-c60e-462c-a0e0-b04a52014db6","Type":"ContainerDied","Data":"f9ac53dd94110da765c6e0a7aa3285b0028e172b5b934486cb1ffe0a1fd9a2b7"}
Oct 30 22:27:22.351366 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:22.351341    5061 generic.go:356] "Generic (PLEG): container finished" podID="1f8d9486-fb2f-491a-a8d7-c23c21bd116a" containerID="457c372be65206f6c139509e0f4926caeefaf8a0848f0b8831e44587c273745f" exitCode=0
Oct 30 22:27:22.351506 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:22.351382    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-14af5c/distinct-w6-14af5c-kserve-7c7cb9d69c-hljl9"
Oct 30 22:27:22.351506 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:22.351415    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-14af5c/distinct-w6-14af5c-kserve-7c7cb9d69c-hljl9" event={"ID":"1f8d9486-fb2f-491a-a8d7-c23c21bd116a","Type":"ContainerDied","Data":"457c372be65206f6c139509e0f4926caeefaf8a0848f0b8831e44587c273745f"}
Oct 30 22:27:22.351823 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:22.351796    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-14af5c/distinct-w6-14af5c-kserve-7c7cb9d69c-hljl9" event={"ID":"1f8d9486-fb2f-491a-a8d7-c23c21bd116a","Type":"ContainerDied","Data":"f7779572a148421ec3b4fa477a28a66f3321d09169a0e136211c8b5089979e38"}
Oct 30 22:27:22.353038 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:22.353020    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-14af5c/simulator-w6-14af5c-kserve-85b8bd96d7-wlcb9"
Oct 30 22:27:22.353164 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:22.353020    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-14af5c/simulator-w6-14af5c-kserve-85b8bd96d7-wlcb9" event={"ID":"39766c7d-5f1d-42db-8920-c0321cd12c99","Type":"ContainerDied","Data":"5ff6dfd897ec36965388217906652f2a8c6347d1b0aa2a64f19b0a947cf04a72"}
Oct 30 22:27:22.362195 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:22.362177    5061 scope.go:122] "RemoveContainer" containerID="b1b2212c3c7180a7832363d0cc2347107f1744364b4e0b1561dd4a8c2fd722cf"
Oct 30 22:27:22.370939 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:22.370922    5061 scope.go:122] "RemoveContainer" containerID="457c372be65206f6c139509e0f4926caeefaf8a0848f0b8831e44587c273745f"
Oct 30 22:27:22.379761 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:22.379739    5061 scope.go:122] "RemoveContainer" containerID="457c372be65206f6c139509e0f4926caeefaf8a0848f0b8831e44587c273745f"
Oct 30 22:27:22.379983 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:27:22.379966    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"457c372be65206f6c139509e0f4926caeefaf8a0848f0b8831e44587c273745f\": container with ID starting with 457c372be65206f6c139509e0f4926caeefaf8a0848f0b8831e44587c273745f not found: ID does not exist" containerID="457c372be65206f6c139509e0f4926caeefaf8a0848f0b8831e44587c273745f"
Oct 30 22:27:22.380039 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:22.379993    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"457c372be65206f6c139509e0f4926caeefaf8a0848f0b8831e44587c273745f"} err="failed to get container status \"457c372be65206f6c139509e0f4926caeefaf8a0848f0b8831e44587c273745f\": rpc error: code = NotFound desc = could not find container \"457c372be65206f6c139509e0f4926caeefaf8a0848f0b8831e44587c273745f\": container with ID starting with 457c372be65206f6c139509e0f4926caeefaf8a0848f0b8831e44587c273745f not found: ID does not exist"
Oct 30 22:27:22.380039 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:22.380009    5061 scope.go:122] "RemoveContainer" containerID="4742be0ecaf01e860e10e4751e0a2620e7d66623622e6cc0756908c05a65b260"
Oct 30 22:27:22.504053 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:22.504030    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-14af5c/simulator-w6-14af5c-kserve-85b8bd96d7-wlcb9"]
Oct 30 22:27:22.551329 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:22.551303    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-14af5c/simulator-w6-14af5c-kserve-85b8bd96d7-wlcb9"]
Oct 30 22:27:22.655228 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:22.655154    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-14af5c/premium-w6-14af5c-kserve-6549647f4-f4gwg"]
Oct 30 22:27:22.659352 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:22.659320    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-14af5c/premium-w6-14af5c-kserve-6549647f4-f4gwg"]
Oct 30 22:27:22.762021 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:22.761992    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-14af5c/distinct-2-w6-14af5c-kserve-58d554bdb6-r5r6j"]
Oct 30 22:27:22.766055 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:22.766029    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-14af5c/distinct-2-w6-14af5c-kserve-58d554bdb6-r5r6j"]
Oct 30 22:27:22.871470 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:22.871440    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-14af5c/distinct-w6-14af5c-kserve-7c7cb9d69c-hljl9"]
Oct 30 22:27:22.875032 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:22.875003    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-14af5c/distinct-w6-14af5c-kserve-7c7cb9d69c-hljl9"]
Oct 30 22:27:24.528379 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:24.528344    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f8d9486-fb2f-491a-a8d7-c23c21bd116a" path="/var/lib/kubelet/pods/1f8d9486-fb2f-491a-a8d7-c23c21bd116a/volumes"
Oct 30 22:27:24.528913 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:24.528898    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39766c7d-5f1d-42db-8920-c0321cd12c99" path="/var/lib/kubelet/pods/39766c7d-5f1d-42db-8920-c0321cd12c99/volumes"
Oct 30 22:27:24.529411 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:24.529393    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="990f4e94-708e-45f3-b2ad-01cd50b3257b" path="/var/lib/kubelet/pods/990f4e94-708e-45f3-b2ad-01cd50b3257b/volumes"
Oct 30 22:27:24.529862 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:24.529848    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b42ee01-c60e-462c-a0e0-b04a52014db6" path="/var/lib/kubelet/pods/9b42ee01-c60e-462c-a0e0-b04a52014db6/volumes"
Oct 30 22:27:24.667299 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:27:24.667265    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 22:27:26.923578 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:26.923515    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-sclrt_5e74071e-0462-42b5-a05c-d90bb638478a/console-operator/2.log"
Oct 30 22:27:26.967401 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:26.967374    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-sclrt_5e74071e-0462-42b5-a05c-d90bb638478a/console-operator/2.log"
Oct 30 22:27:29.092179 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:29.092147    5061 scope.go:122] "RemoveContainer" containerID="6a8a0277544b978e7ebb53d6dafedcdbf49237cd4c56520a3904176fa032b365"
Oct 30 22:27:29.101752 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:29.101676    5061 scope.go:122] "RemoveContainer" containerID="0acb61a5a99201b677ce533536f7ca512d81052d1ff57fce086ea4eca0d1f852"
Oct 30 22:27:29.110308 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:29.110280    5061 scope.go:122] "RemoveContainer" containerID="5edae1f59bc47fb50fef948d15ed8baff418955014b67f38a74cc1f224b539de"
Oct 30 22:27:30.700209 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:30.700174    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-14af5c-85d68c5546-r4slw" podUID="6977b446-b5f9-4f27-8d41-a4942379c9b2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:27:37.002138 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:37.002101    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84b1935f-m69v4"]
Oct 30 22:27:37.002670 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:37.002650    5061 cpu_manager.go:414] "containerMap: removing container" podUID="de890e38-beb5-4920-9525-66f7d57d2216" containerName="revoke-keys"
Oct 30 22:27:37.002670 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:37.002665    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de890e38-beb5-4920-9525-66f7d57d2216" containerName="revoke-keys"
Oct 30 22:27:37.002760 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:37.002688    5061 cpu_manager.go:414] "containerMap: removing container" podUID="39766c7d-5f1d-42db-8920-c0321cd12c99" containerName="main"
Oct 30 22:27:37.002760 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:37.002693    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39766c7d-5f1d-42db-8920-c0321cd12c99" containerName="main"
Oct 30 22:27:37.002760 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:37.002709    5061 cpu_manager.go:414] "containerMap: removing container" podUID="1f8d9486-fb2f-491a-a8d7-c23c21bd116a" containerName="main"
Oct 30 22:27:37.002760 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:37.002715    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f8d9486-fb2f-491a-a8d7-c23c21bd116a" containerName="main"
Oct 30 22:27:37.002760 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:37.002722    5061 cpu_manager.go:414] "containerMap: removing container" podUID="ea11c8ac-4a65-463d-8956-739f3ae20611" containerName="istio-proxy"
Oct 30 22:27:37.002760 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:37.002727    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea11c8ac-4a65-463d-8956-739f3ae20611" containerName="istio-proxy"
Oct 30 22:27:37.002760 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:37.002739    5061 cpu_manager.go:414] "containerMap: removing container" podUID="990f4e94-708e-45f3-b2ad-01cd50b3257b" containerName="main"
Oct 30 22:27:37.002760 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:37.002744    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="990f4e94-708e-45f3-b2ad-01cd50b3257b" containerName="main"
Oct 30 22:27:37.002760 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:37.002757    5061 cpu_manager.go:414] "containerMap: removing container" podUID="9b42ee01-c60e-462c-a0e0-b04a52014db6" containerName="main"
Oct 30 22:27:37.002760 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:37.002762    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b42ee01-c60e-462c-a0e0-b04a52014db6" containerName="main"
Oct 30 22:27:37.003178 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:37.002852    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="990f4e94-708e-45f3-b2ad-01cd50b3257b" containerName="main"
Oct 30 22:27:37.003178 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:37.002868    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b42ee01-c60e-462c-a0e0-b04a52014db6" containerName="main"
Oct 30 22:27:37.003178 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:37.002879    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="de890e38-beb5-4920-9525-66f7d57d2216" containerName="revoke-keys"
Oct 30 22:27:37.003178 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:37.002891    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="39766c7d-5f1d-42db-8920-c0321cd12c99" containerName="main"
Oct 30 22:27:37.003178 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:37.002903    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f8d9486-fb2f-491a-a8d7-c23c21bd116a" containerName="main"
Oct 30 22:27:37.003178 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:37.002911    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea11c8ac-4a65-463d-8956-739f3ae20611" containerName="istio-proxy"
Oct 30 22:27:37.008201 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:37.008178    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84b1935f-m69v4"
Oct 30 22:27:37.053294 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:37.053265    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31c8b66a-n4cx4"]
Oct 30 22:27:37.057211 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:37.057185    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31c8b66a-n4cx4"
Oct 30 22:27:37.057628 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:37.057609    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fc81a52-bb17-4dc5-9de2-62ae34ee7917-maas-api-xks-ca\") pod \"maas-api-revoke-sub-84b1935f-m69v4\" (UID: \"9fc81a52-bb17-4dc5-9de2-62ae34ee7917\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84b1935f-m69v4"
Oct 30 22:27:37.057697 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:37.057645    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fc81a52-bb17-4dc5-9de2-62ae34ee7917-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-84b1935f-m69v4\" (UID: \"9fc81a52-bb17-4dc5-9de2-62ae34ee7917\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84b1935f-m69v4"
Oct 30 22:27:37.057697 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:37.057665    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbnrp\" (UniqueName: \"kubernetes.io/projected/9fc81a52-bb17-4dc5-9de2-62ae34ee7917-kube-api-access-tbnrp\") pod \"maas-api-revoke-sub-84b1935f-m69v4\" (UID: \"9fc81a52-bb17-4dc5-9de2-62ae34ee7917\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84b1935f-m69v4"
Oct 30 22:27:37.097796 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:37.097760    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:27:37.100768 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:37.100739    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84b1935f-m69v4"]
Oct 30 22:27:37.105920 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:37.105896    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gdmfg\""
Oct 30 22:27:37.115087 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:37.115060    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31c8b66a-n4cx4"]
Oct 30 22:27:37.158818 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:37.158784    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fc81a52-bb17-4dc5-9de2-62ae34ee7917-maas-api-xks-ca\") pod \"maas-api-revoke-sub-84b1935f-m69v4\" (UID: \"9fc81a52-bb17-4dc5-9de2-62ae34ee7917\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84b1935f-m69v4"
Oct 30 22:27:37.159022 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:37.158827    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fc81a52-bb17-4dc5-9de2-62ae34ee7917-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-84b1935f-m69v4\" (UID: \"9fc81a52-bb17-4dc5-9de2-62ae34ee7917\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84b1935f-m69v4"
Oct 30 22:27:37.159022 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:37.158846    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbnrp\" (UniqueName: \"kubernetes.io/projected/9fc81a52-bb17-4dc5-9de2-62ae34ee7917-kube-api-access-tbnrp\") pod \"maas-api-revoke-sub-84b1935f-m69v4\" (UID: \"9fc81a52-bb17-4dc5-9de2-62ae34ee7917\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84b1935f-m69v4"
Oct 30 22:27:37.159022 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:37.158879    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dz52\" (UniqueName: \"kubernetes.io/projected/44d23433-b77d-49d4-a5a3-a969f2d0507d-kube-api-access-2dz52\") pod \"maas-api-revoke-sub-31c8b66a-n4cx4\" (UID: \"44d23433-b77d-49d4-a5a3-a969f2d0507d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31c8b66a-n4cx4"
Oct 30 22:27:37.159022 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:37.158922    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44d23433-b77d-49d4-a5a3-a969f2d0507d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-31c8b66a-n4cx4\" (UID: \"44d23433-b77d-49d4-a5a3-a969f2d0507d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31c8b66a-n4cx4"
Oct 30 22:27:37.159022 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:37.158944    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44d23433-b77d-49d4-a5a3-a969f2d0507d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-31c8b66a-n4cx4\" (UID: \"44d23433-b77d-49d4-a5a3-a969f2d0507d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31c8b66a-n4cx4"
Oct 30 22:27:37.159613 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:37.159591    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fc81a52-bb17-4dc5-9de2-62ae34ee7917-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-84b1935f-m69v4\" (UID: \"9fc81a52-bb17-4dc5-9de2-62ae34ee7917\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84b1935f-m69v4"
Oct 30 22:27:37.161338 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:37.161309    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fc81a52-bb17-4dc5-9de2-62ae34ee7917-maas-api-xks-ca\") pod \"maas-api-revoke-sub-84b1935f-m69v4\" (UID: \"9fc81a52-bb17-4dc5-9de2-62ae34ee7917\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84b1935f-m69v4"
Oct 30 22:27:37.213089 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:37.213060    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbnrp\" (UniqueName: \"kubernetes.io/projected/9fc81a52-bb17-4dc5-9de2-62ae34ee7917-kube-api-access-tbnrp\") pod \"maas-api-revoke-sub-84b1935f-m69v4\" (UID: \"9fc81a52-bb17-4dc5-9de2-62ae34ee7917\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84b1935f-m69v4"
Oct 30 22:27:37.260634 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:37.260559    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dz52\" (UniqueName: \"kubernetes.io/projected/44d23433-b77d-49d4-a5a3-a969f2d0507d-kube-api-access-2dz52\") pod \"maas-api-revoke-sub-31c8b66a-n4cx4\" (UID: \"44d23433-b77d-49d4-a5a3-a969f2d0507d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31c8b66a-n4cx4"
Oct 30 22:27:37.260634 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:37.260610    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44d23433-b77d-49d4-a5a3-a969f2d0507d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-31c8b66a-n4cx4\" (UID: \"44d23433-b77d-49d4-a5a3-a969f2d0507d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31c8b66a-n4cx4"
Oct 30 22:27:37.260634 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:37.260635    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44d23433-b77d-49d4-a5a3-a969f2d0507d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-31c8b66a-n4cx4\" (UID: \"44d23433-b77d-49d4-a5a3-a969f2d0507d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31c8b66a-n4cx4"
Oct 30 22:27:37.261327 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:37.261299    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44d23433-b77d-49d4-a5a3-a969f2d0507d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-31c8b66a-n4cx4\" (UID: \"44d23433-b77d-49d4-a5a3-a969f2d0507d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31c8b66a-n4cx4"
Oct 30 22:27:37.262855 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:37.262835    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44d23433-b77d-49d4-a5a3-a969f2d0507d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-31c8b66a-n4cx4\" (UID: \"44d23433-b77d-49d4-a5a3-a969f2d0507d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31c8b66a-n4cx4"
Oct 30 22:27:37.315273 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:37.315243    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dz52\" (UniqueName: \"kubernetes.io/projected/44d23433-b77d-49d4-a5a3-a969f2d0507d-kube-api-access-2dz52\") pod \"maas-api-revoke-sub-31c8b66a-n4cx4\" (UID: \"44d23433-b77d-49d4-a5a3-a969f2d0507d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31c8b66a-n4cx4"
Oct 30 22:27:37.322083 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:37.322060    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84b1935f-m69v4"
Oct 30 22:27:37.372647 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:37.372610    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31c8b66a-n4cx4"
Oct 30 22:27:37.689491 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:37.689461    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84b1935f-m69v4"]
Oct 30 22:27:37.690816 ip-10-0-130-60 kubenswrapper[5061]: W1030 22:27:37.690790    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fc81a52_bb17_4dc5_9de2_62ae34ee7917.slice/crio-9a5994c52a709b1266a69f121e1ef62d42a7f0aa9932c1f3eb7e6a260d5a99d0 WatchSource:0}: Error finding container 9a5994c52a709b1266a69f121e1ef62d42a7f0aa9932c1f3eb7e6a260d5a99d0: Status 404 returned error can't find the container with id 9a5994c52a709b1266a69f121e1ef62d42a7f0aa9932c1f3eb7e6a260d5a99d0
Oct 30 22:27:37.798066 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:37.798030    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31c8b66a-n4cx4"]
Oct 30 22:27:37.798066 ip-10-0-130-60 kubenswrapper[5061]: W1030 22:27:37.798054    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44d23433_b77d_49d4_a5a3_a969f2d0507d.slice/crio-ea0e061fdbb78f14e8d34e83450eeba0fb1dd8c8f565feca479803a8f3d40402 WatchSource:0}: Error finding container ea0e061fdbb78f14e8d34e83450eeba0fb1dd8c8f565feca479803a8f3d40402: Status 404 returned error can't find the container with id ea0e061fdbb78f14e8d34e83450eeba0fb1dd8c8f565feca479803a8f3d40402
Oct 30 22:27:38.325260 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:38.325233    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-14af5c-85d68c5546-r4slw"
Oct 30 22:27:38.379011 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:38.378983    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6977b446-b5f9-4f27-8d41-a4942379c9b2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6977b446-b5f9-4f27-8d41-a4942379c9b2-plugins-config-volume\") pod \"6977b446-b5f9-4f27-8d41-a4942379c9b2\" (UID: \"6977b446-b5f9-4f27-8d41-a4942379c9b2\") "
Oct 30 22:27:38.379172 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:38.379022    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6977b446-b5f9-4f27-8d41-a4942379c9b2-kube-api-access-fhjc6\" (UniqueName: \"kubernetes.io/projected/6977b446-b5f9-4f27-8d41-a4942379c9b2-kube-api-access-fhjc6\") pod \"6977b446-b5f9-4f27-8d41-a4942379c9b2\" (UID: \"6977b446-b5f9-4f27-8d41-a4942379c9b2\") "
Oct 30 22:27:38.379347 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:38.379326    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6977b446-b5f9-4f27-8d41-a4942379c9b2-plugins-config-volume" pod "6977b446-b5f9-4f27-8d41-a4942379c9b2" (UID: "6977b446-b5f9-4f27-8d41-a4942379c9b2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:27:38.381229 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:38.381209    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6977b446-b5f9-4f27-8d41-a4942379c9b2-kube-api-access-fhjc6" pod "6977b446-b5f9-4f27-8d41-a4942379c9b2" (UID: "6977b446-b5f9-4f27-8d41-a4942379c9b2"). InnerVolumeSpecName "kube-api-access-fhjc6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:27:38.469161 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:38.469118    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31c8b66a-n4cx4" event={"ID":"44d23433-b77d-49d4-a5a3-a969f2d0507d","Type":"ContainerStarted","Data":"d438dcc9f5fa071c157231d56ac38fc7735606d45fe5837862512a2178bc2b80"}
Oct 30 22:27:38.469365 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:38.469225    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31c8b66a-n4cx4" event={"ID":"44d23433-b77d-49d4-a5a3-a969f2d0507d","Type":"ContainerStarted","Data":"ea0e061fdbb78f14e8d34e83450eeba0fb1dd8c8f565feca479803a8f3d40402"}
Oct 30 22:27:38.470943 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:38.470916    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84b1935f-m69v4" event={"ID":"9fc81a52-bb17-4dc5-9de2-62ae34ee7917","Type":"ContainerStarted","Data":"134457dc3ce8eba0244b97d9913e2c2ac75d58ca030280fdc9e4da116bd3c3ea"}
Oct 30 22:27:38.471060 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:38.470964    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84b1935f-m69v4" event={"ID":"9fc81a52-bb17-4dc5-9de2-62ae34ee7917","Type":"ContainerStarted","Data":"9a5994c52a709b1266a69f121e1ef62d42a7f0aa9932c1f3eb7e6a260d5a99d0"}
Oct 30 22:27:38.472318 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:38.472291    5061 generic.go:356] "Generic (PLEG): container finished" podID="6977b446-b5f9-4f27-8d41-a4942379c9b2" containerID="86d8a666c1779dda82bfa30c16d519a8d51af7e4b1517f6f22318facc31637d1" exitCode=0
Oct 30 22:27:38.472429 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:38.472330    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-14af5c-85d68c5546-r4slw"
Oct 30 22:27:38.472429 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:38.472367    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-14af5c-85d68c5546-r4slw" event={"ID":"6977b446-b5f9-4f27-8d41-a4942379c9b2","Type":"ContainerDied","Data":"86d8a666c1779dda82bfa30c16d519a8d51af7e4b1517f6f22318facc31637d1"}
Oct 30 22:27:38.472429 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:38.472421    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-14af5c-85d68c5546-r4slw" event={"ID":"6977b446-b5f9-4f27-8d41-a4942379c9b2","Type":"ContainerDied","Data":"7ae05b3119e8374fef5f9e038d0727a741f7d0d1b29a2826c0d0ebf5662fa48c"}
Oct 30 22:27:38.472640 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:38.472451    5061 scope.go:122] "RemoveContainer" containerID="86d8a666c1779dda82bfa30c16d519a8d51af7e4b1517f6f22318facc31637d1"
Oct 30 22:27:38.480053 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:38.480018    5061 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6977b446-b5f9-4f27-8d41-a4942379c9b2-plugins-config-volume\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:38.480167 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:38.480063    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhjc6\" (UniqueName: \"kubernetes.io/projected/6977b446-b5f9-4f27-8d41-a4942379c9b2-kube-api-access-fhjc6\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:38.483664 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:38.483645    5061 scope.go:122] "RemoveContainer" containerID="86d8a666c1779dda82bfa30c16d519a8d51af7e4b1517f6f22318facc31637d1"
Oct 30 22:27:38.483949 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:27:38.483928    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86d8a666c1779dda82bfa30c16d519a8d51af7e4b1517f6f22318facc31637d1\": container with ID starting with 86d8a666c1779dda82bfa30c16d519a8d51af7e4b1517f6f22318facc31637d1 not found: ID does not exist" containerID="86d8a666c1779dda82bfa30c16d519a8d51af7e4b1517f6f22318facc31637d1"
Oct 30 22:27:38.483999 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:38.483957    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86d8a666c1779dda82bfa30c16d519a8d51af7e4b1517f6f22318facc31637d1"} err="failed to get container status \"86d8a666c1779dda82bfa30c16d519a8d51af7e4b1517f6f22318facc31637d1\": rpc error: code = NotFound desc = could not find container \"86d8a666c1779dda82bfa30c16d519a8d51af7e4b1517f6f22318facc31637d1\": container with ID starting with 86d8a666c1779dda82bfa30c16d519a8d51af7e4b1517f6f22318facc31637d1 not found: ID does not exist"
Oct 30 22:27:38.580290 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:38.580189    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31c8b66a-n4cx4" podStartSLOduration=1.580176594 podStartE2EDuration="1.580176594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:27:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:27:38.577634444 +0000 UTC m=+2712.190404201" watchObservedRunningTime="2026-10-30 22:27:38.580176594 +0000 UTC m=+2712.192946369"
Oct 30 22:27:38.688619 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:38.688570    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84b1935f-m69v4" podStartSLOduration=2.68855671 podStartE2EDuration="2.68855671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:27:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:27:38.6850863 +0000 UTC m=+2712.297856059" watchObservedRunningTime="2026-10-30 22:27:38.68855671 +0000 UTC m=+2712.301326468"
Oct 30 22:27:38.791481 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:38.791450    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-14af5c-85d68c5546-r4slw"]
Oct 30 22:27:38.795231 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:38.795203    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-14af5c-85d68c5546-r4slw"]
Oct 30 22:27:39.481603 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:39.481499    5061 generic.go:356] "Generic (PLEG): container finished" podID="44d23433-b77d-49d4-a5a3-a969f2d0507d" containerID="d438dcc9f5fa071c157231d56ac38fc7735606d45fe5837862512a2178bc2b80" exitCode=0
Oct 30 22:27:39.481603 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:39.481565    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31c8b66a-n4cx4" event={"ID":"44d23433-b77d-49d4-a5a3-a969f2d0507d","Type":"ContainerDied","Data":"d438dcc9f5fa071c157231d56ac38fc7735606d45fe5837862512a2178bc2b80"}
Oct 30 22:27:39.483196 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:39.483169    5061 generic.go:356] "Generic (PLEG): container finished" podID="9fc81a52-bb17-4dc5-9de2-62ae34ee7917" containerID="134457dc3ce8eba0244b97d9913e2c2ac75d58ca030280fdc9e4da116bd3c3ea" exitCode=0
Oct 30 22:27:39.483322 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:39.483245    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84b1935f-m69v4" event={"ID":"9fc81a52-bb17-4dc5-9de2-62ae34ee7917","Type":"ContainerDied","Data":"134457dc3ce8eba0244b97d9913e2c2ac75d58ca030280fdc9e4da116bd3c3ea"}
Oct 30 22:27:40.532214 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:40.532170    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6977b446-b5f9-4f27-8d41-a4942379c9b2" path="/var/lib/kubelet/pods/6977b446-b5f9-4f27-8d41-a4942379c9b2/volumes"
Oct 30 22:27:40.676652 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:40.676626    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31c8b66a-n4cx4"
Oct 30 22:27:40.678853 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:40.678836    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84b1935f-m69v4"
Oct 30 22:27:40.806996 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:40.806949    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fc81a52-bb17-4dc5-9de2-62ae34ee7917-kube-api-access-tbnrp\" (UniqueName: \"kubernetes.io/projected/9fc81a52-bb17-4dc5-9de2-62ae34ee7917-kube-api-access-tbnrp\") pod \"9fc81a52-bb17-4dc5-9de2-62ae34ee7917\" (UID: \"9fc81a52-bb17-4dc5-9de2-62ae34ee7917\") "
Oct 30 22:27:40.807196 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:40.807022    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9fc81a52-bb17-4dc5-9de2-62ae34ee7917-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fc81a52-bb17-4dc5-9de2-62ae34ee7917-maas-api-xks-ca\") pod \"9fc81a52-bb17-4dc5-9de2-62ae34ee7917\" (UID: \"9fc81a52-bb17-4dc5-9de2-62ae34ee7917\") "
Oct 30 22:27:40.807196 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:40.807077    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/44d23433-b77d-49d4-a5a3-a969f2d0507d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44d23433-b77d-49d4-a5a3-a969f2d0507d-maas-api-xks-ca\") pod \"44d23433-b77d-49d4-a5a3-a969f2d0507d\" (UID: \"44d23433-b77d-49d4-a5a3-a969f2d0507d\") "
Oct 30 22:27:40.807196 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:40.807130    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44d23433-b77d-49d4-a5a3-a969f2d0507d-kube-api-access-2dz52\" (UniqueName: \"kubernetes.io/projected/44d23433-b77d-49d4-a5a3-a969f2d0507d-kube-api-access-2dz52\") pod \"44d23433-b77d-49d4-a5a3-a969f2d0507d\" (UID: \"44d23433-b77d-49d4-a5a3-a969f2d0507d\") "
Oct 30 22:27:40.807196 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:40.807171    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44d23433-b77d-49d4-a5a3-a969f2d0507d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44d23433-b77d-49d4-a5a3-a969f2d0507d-maas-api-openshift-ca\") pod \"44d23433-b77d-49d4-a5a3-a969f2d0507d\" (UID: \"44d23433-b77d-49d4-a5a3-a969f2d0507d\") "
Oct 30 22:27:40.807450 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:40.807224    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fc81a52-bb17-4dc5-9de2-62ae34ee7917-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fc81a52-bb17-4dc5-9de2-62ae34ee7917-maas-api-openshift-ca\") pod \"9fc81a52-bb17-4dc5-9de2-62ae34ee7917\" (UID: \"9fc81a52-bb17-4dc5-9de2-62ae34ee7917\") "
Oct 30 22:27:40.807797 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:40.807769    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44d23433-b77d-49d4-a5a3-a969f2d0507d-maas-api-openshift-ca" pod "44d23433-b77d-49d4-a5a3-a969f2d0507d" (UID: "44d23433-b77d-49d4-a5a3-a969f2d0507d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:27:40.807914 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:40.807843    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fc81a52-bb17-4dc5-9de2-62ae34ee7917-maas-api-openshift-ca" pod "9fc81a52-bb17-4dc5-9de2-62ae34ee7917" (UID: "9fc81a52-bb17-4dc5-9de2-62ae34ee7917"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:27:40.809342 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:40.809308    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fc81a52-bb17-4dc5-9de2-62ae34ee7917-kube-api-access-tbnrp" pod "9fc81a52-bb17-4dc5-9de2-62ae34ee7917" (UID: "9fc81a52-bb17-4dc5-9de2-62ae34ee7917"). InnerVolumeSpecName "kube-api-access-tbnrp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:27:40.809844 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:40.809813    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fc81a52-bb17-4dc5-9de2-62ae34ee7917-maas-api-xks-ca" pod "9fc81a52-bb17-4dc5-9de2-62ae34ee7917" (UID: "9fc81a52-bb17-4dc5-9de2-62ae34ee7917"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:27:40.809950 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:40.809850    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44d23433-b77d-49d4-a5a3-a969f2d0507d-kube-api-access-2dz52" pod "44d23433-b77d-49d4-a5a3-a969f2d0507d" (UID: "44d23433-b77d-49d4-a5a3-a969f2d0507d"). InnerVolumeSpecName "kube-api-access-2dz52". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:27:40.810098 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:40.810077    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44d23433-b77d-49d4-a5a3-a969f2d0507d-maas-api-xks-ca" pod "44d23433-b77d-49d4-a5a3-a969f2d0507d" (UID: "44d23433-b77d-49d4-a5a3-a969f2d0507d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:27:40.908862 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:40.908828    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fc81a52-bb17-4dc5-9de2-62ae34ee7917-maas-api-openshift-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:40.908862 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:40.908857    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbnrp\" (UniqueName: \"kubernetes.io/projected/9fc81a52-bb17-4dc5-9de2-62ae34ee7917-kube-api-access-tbnrp\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:40.909122 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:40.908881    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fc81a52-bb17-4dc5-9de2-62ae34ee7917-maas-api-xks-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:40.909122 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:40.908894    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44d23433-b77d-49d4-a5a3-a969f2d0507d-maas-api-xks-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:40.909122 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:40.908904    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dz52\" (UniqueName: \"kubernetes.io/projected/44d23433-b77d-49d4-a5a3-a969f2d0507d-kube-api-access-2dz52\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:40.909122 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:40.908913    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44d23433-b77d-49d4-a5a3-a969f2d0507d-maas-api-openshift-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:41.499024 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:41.498991    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31c8b66a-n4cx4"
Oct 30 22:27:41.499203 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:41.498993    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31c8b66a-n4cx4" event={"ID":"44d23433-b77d-49d4-a5a3-a969f2d0507d","Type":"ContainerDied","Data":"ea0e061fdbb78f14e8d34e83450eeba0fb1dd8c8f565feca479803a8f3d40402"}
Oct 30 22:27:41.499203 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:41.499143    5061 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ea0e061fdbb78f14e8d34e83450eeba0fb1dd8c8f565feca479803a8f3d40402"
Oct 30 22:27:41.500226 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:41.500208    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84b1935f-m69v4"
Oct 30 22:27:41.500359 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:41.500236    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84b1935f-m69v4" event={"ID":"9fc81a52-bb17-4dc5-9de2-62ae34ee7917","Type":"ContainerDied","Data":"9a5994c52a709b1266a69f121e1ef62d42a7f0aa9932c1f3eb7e6a260d5a99d0"}
Oct 30 22:27:41.500359 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:27:41.500267    5061 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9a5994c52a709b1266a69f121e1ef62d42a7f0aa9932c1f3eb7e6a260d5a99d0"
Oct 30 22:27:54.667818 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:27:54.667773    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 22:28:10.885492 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:10.885415    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-14af5c-bd596f55c-f4z8t"]
Oct 30 22:28:10.887887 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:10.885652    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-14af5c-bd596f55c-f4z8t" podUID="4646779e-fc85-4860-adcf-9855d2d26fb1" containerName="maas-api" containerID="cri-o://f9d1f8f098f59de29fa76e16823e6a77e792a849e274d8366cb07eb7cb4e19be" gracePeriod=30
Oct 30 22:28:11.168650 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:11.168621    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-14af5c-bd596f55c-f4z8t"
Oct 30 22:28:11.235776 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:11.235747    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4646779e-fc85-4860-adcf-9855d2d26fb1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4646779e-fc85-4860-adcf-9855d2d26fb1-metrics-certs\") pod \"4646779e-fc85-4860-adcf-9855d2d26fb1\" (UID: \"4646779e-fc85-4860-adcf-9855d2d26fb1\") "
Oct 30 22:28:11.235940 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:11.235807    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4646779e-fc85-4860-adcf-9855d2d26fb1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4646779e-fc85-4860-adcf-9855d2d26fb1-maas-api-tls\") pod \"4646779e-fc85-4860-adcf-9855d2d26fb1\" (UID: \"4646779e-fc85-4860-adcf-9855d2d26fb1\") "
Oct 30 22:28:11.235940 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:11.235900    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4646779e-fc85-4860-adcf-9855d2d26fb1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4646779e-fc85-4860-adcf-9855d2d26fb1-openshift-service-ca\") pod \"4646779e-fc85-4860-adcf-9855d2d26fb1\" (UID: \"4646779e-fc85-4860-adcf-9855d2d26fb1\") "
Oct 30 22:28:11.235940 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:11.235932    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4646779e-fc85-4860-adcf-9855d2d26fb1-kube-api-access-qc8g2\" (UniqueName: \"kubernetes.io/projected/4646779e-fc85-4860-adcf-9855d2d26fb1-kube-api-access-qc8g2\") pod \"4646779e-fc85-4860-adcf-9855d2d26fb1\" (UID: \"4646779e-fc85-4860-adcf-9855d2d26fb1\") "
Oct 30 22:28:11.236318 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:11.236285    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4646779e-fc85-4860-adcf-9855d2d26fb1-openshift-service-ca" pod "4646779e-fc85-4860-adcf-9855d2d26fb1" (UID: "4646779e-fc85-4860-adcf-9855d2d26fb1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:28:11.237997 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:11.237974    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4646779e-fc85-4860-adcf-9855d2d26fb1-maas-api-tls" pod "4646779e-fc85-4860-adcf-9855d2d26fb1" (UID: "4646779e-fc85-4860-adcf-9855d2d26fb1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:28:11.238091 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:11.237994    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4646779e-fc85-4860-adcf-9855d2d26fb1-metrics-certs" pod "4646779e-fc85-4860-adcf-9855d2d26fb1" (UID: "4646779e-fc85-4860-adcf-9855d2d26fb1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:28:11.238091 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:11.238055    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4646779e-fc85-4860-adcf-9855d2d26fb1-kube-api-access-qc8g2" pod "4646779e-fc85-4860-adcf-9855d2d26fb1" (UID: "4646779e-fc85-4860-adcf-9855d2d26fb1"). InnerVolumeSpecName "kube-api-access-qc8g2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:28:11.337597 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:11.337565    5061 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4646779e-fc85-4860-adcf-9855d2d26fb1-openshift-service-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:11.337597 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:11.337595    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qc8g2\" (UniqueName: \"kubernetes.io/projected/4646779e-fc85-4860-adcf-9855d2d26fb1-kube-api-access-qc8g2\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:11.337796 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:11.337606    5061 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4646779e-fc85-4860-adcf-9855d2d26fb1-metrics-certs\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:11.337796 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:11.337615    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4646779e-fc85-4860-adcf-9855d2d26fb1-maas-api-tls\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:11.689310 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:11.689277    5061 generic.go:356] "Generic (PLEG): container finished" podID="4646779e-fc85-4860-adcf-9855d2d26fb1" containerID="f9d1f8f098f59de29fa76e16823e6a77e792a849e274d8366cb07eb7cb4e19be" exitCode=0
Oct 30 22:28:11.689495 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:11.689330    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-14af5c-bd596f55c-f4z8t" event={"ID":"4646779e-fc85-4860-adcf-9855d2d26fb1","Type":"ContainerDied","Data":"f9d1f8f098f59de29fa76e16823e6a77e792a849e274d8366cb07eb7cb4e19be"}
Oct 30 22:28:11.689495 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:11.689334    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-14af5c-bd596f55c-f4z8t"
Oct 30 22:28:11.689495 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:11.689369    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-14af5c-bd596f55c-f4z8t" event={"ID":"4646779e-fc85-4860-adcf-9855d2d26fb1","Type":"ContainerDied","Data":"e2bfe1093bf39b6079c816f77109a76c143e595328f461ea11efdb3ca78d71db"}
Oct 30 22:28:11.689495 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:11.689392    5061 scope.go:122] "RemoveContainer" containerID="f9d1f8f098f59de29fa76e16823e6a77e792a849e274d8366cb07eb7cb4e19be"
Oct 30 22:28:11.700255 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:11.700231    5061 scope.go:122] "RemoveContainer" containerID="f9d1f8f098f59de29fa76e16823e6a77e792a849e274d8366cb07eb7cb4e19be"
Oct 30 22:28:11.700544 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:28:11.700511    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9d1f8f098f59de29fa76e16823e6a77e792a849e274d8366cb07eb7cb4e19be\": container with ID starting with f9d1f8f098f59de29fa76e16823e6a77e792a849e274d8366cb07eb7cb4e19be not found: ID does not exist" containerID="f9d1f8f098f59de29fa76e16823e6a77e792a849e274d8366cb07eb7cb4e19be"
Oct 30 22:28:11.700594 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:11.700560    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9d1f8f098f59de29fa76e16823e6a77e792a849e274d8366cb07eb7cb4e19be"} err="failed to get container status \"f9d1f8f098f59de29fa76e16823e6a77e792a849e274d8366cb07eb7cb4e19be\": rpc error: code = NotFound desc = could not find container \"f9d1f8f098f59de29fa76e16823e6a77e792a849e274d8366cb07eb7cb4e19be\": container with ID starting with f9d1f8f098f59de29fa76e16823e6a77e792a849e274d8366cb07eb7cb4e19be not found: ID does not exist"
Oct 30 22:28:11.855578 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:11.855235    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-14af5c-bd596f55c-f4z8t"]
Oct 30 22:28:11.897083 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:11.897052    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-14af5c-bd596f55c-f4z8t"]
Oct 30 22:28:12.528506 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:12.528473    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4646779e-fc85-4860-adcf-9855d2d26fb1" path="/var/lib/kubelet/pods/4646779e-fc85-4860-adcf-9855d2d26fb1/volumes"
Oct 30 22:28:13.003993 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:13.003953    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-14af5c-fk9jt"]
Oct 30 22:28:13.049945 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:13.049913    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-14af5c-fk9jt"]
Oct 30 22:28:14.528101 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:14.528069    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de890e38-beb5-4920-9525-66f7d57d2216" path="/var/lib/kubelet/pods/de890e38-beb5-4920-9525-66f7d57d2216/volumes"
Oct 30 22:28:15.046836 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:15.046802    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f29a2d08-qzn4l"]
Oct 30 22:28:15.094233 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:15.094201    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2d9b5eb-ws4sd"]
Oct 30 22:28:15.096721 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:15.096702    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f29a2d08-qzn4l"]
Oct 30 22:28:15.099010 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:15.098984    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2d9b5eb-ws4sd"]
Oct 30 22:28:16.527508 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:16.527476    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73bba646-34ed-4c37-8cdb-df7b4ea77e7f" path="/var/lib/kubelet/pods/73bba646-34ed-4c37-8cdb-df7b4ea77e7f/volumes"
Oct 30 22:28:16.527954 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:16.527937    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa64ef6b-8386-4f0b-b741-b77e4b5d4865" path="/var/lib/kubelet/pods/fa64ef6b-8386-4f0b-b741-b77e4b5d4865/volumes"
Oct 30 22:28:20.265405 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:20.265364    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-shwf8"]
Oct 30 22:28:20.265907 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:20.265635    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-shwf8" podUID="9f137aa3-0e26-4526-92b0-2636c979075f" containerName="istio-proxy" containerID="cri-o://c85d056c85a8b6e789e406e2e1db65cd780b651b9239857a721b511f348c1b76" gracePeriod=30
Oct 30 22:28:24.666588 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:28:24.666551    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 22:28:25.517795 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:25.517771    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-shwf8"
Oct 30 22:28:25.618757 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:25.618732    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f137aa3-0e26-4526-92b0-2636c979075f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f137aa3-0e26-4526-92b0-2636c979075f-istiod-ca-cert\") pod \"9f137aa3-0e26-4526-92b0-2636c979075f\" (UID: \"9f137aa3-0e26-4526-92b0-2636c979075f\") "
Oct 30 22:28:25.618887 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:25.618803    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f137aa3-0e26-4526-92b0-2636c979075f-kube-api-access-jsxp6\" (UniqueName: \"kubernetes.io/projected/9f137aa3-0e26-4526-92b0-2636c979075f-kube-api-access-jsxp6\") pod \"9f137aa3-0e26-4526-92b0-2636c979075f\" (UID: \"9f137aa3-0e26-4526-92b0-2636c979075f\") "
Oct 30 22:28:25.618887 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:25.618837    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f137aa3-0e26-4526-92b0-2636c979075f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f137aa3-0e26-4526-92b0-2636c979075f-istio-data\") pod \"9f137aa3-0e26-4526-92b0-2636c979075f\" (UID: \"9f137aa3-0e26-4526-92b0-2636c979075f\") "
Oct 30 22:28:25.618887 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:25.618862    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9f137aa3-0e26-4526-92b0-2636c979075f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f137aa3-0e26-4526-92b0-2636c979075f-istio-podinfo\") pod \"9f137aa3-0e26-4526-92b0-2636c979075f\" (UID: \"9f137aa3-0e26-4526-92b0-2636c979075f\") "
Oct 30 22:28:25.619080 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:25.618889    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f137aa3-0e26-4526-92b0-2636c979075f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f137aa3-0e26-4526-92b0-2636c979075f-istio-envoy\") pod \"9f137aa3-0e26-4526-92b0-2636c979075f\" (UID: \"9f137aa3-0e26-4526-92b0-2636c979075f\") "
Oct 30 22:28:25.619080 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:25.618939    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f137aa3-0e26-4526-92b0-2636c979075f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f137aa3-0e26-4526-92b0-2636c979075f-workload-socket\") pod \"9f137aa3-0e26-4526-92b0-2636c979075f\" (UID: \"9f137aa3-0e26-4526-92b0-2636c979075f\") "
Oct 30 22:28:25.619080 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:25.619012    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f137aa3-0e26-4526-92b0-2636c979075f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f137aa3-0e26-4526-92b0-2636c979075f-credential-socket\") pod \"9f137aa3-0e26-4526-92b0-2636c979075f\" (UID: \"9f137aa3-0e26-4526-92b0-2636c979075f\") "
Oct 30 22:28:25.619080 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:25.619054    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f137aa3-0e26-4526-92b0-2636c979075f-istio-token\" (UniqueName: \"kubernetes.io/projected/9f137aa3-0e26-4526-92b0-2636c979075f-istio-token\") pod \"9f137aa3-0e26-4526-92b0-2636c979075f\" (UID: \"9f137aa3-0e26-4526-92b0-2636c979075f\") "
Oct 30 22:28:25.619310 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:25.619077    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f137aa3-0e26-4526-92b0-2636c979075f-istiod-ca-cert" pod "9f137aa3-0e26-4526-92b0-2636c979075f" (UID: "9f137aa3-0e26-4526-92b0-2636c979075f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:28:25.619310 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:25.619095    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f137aa3-0e26-4526-92b0-2636c979075f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f137aa3-0e26-4526-92b0-2636c979075f-workload-certs\") pod \"9f137aa3-0e26-4526-92b0-2636c979075f\" (UID: \"9f137aa3-0e26-4526-92b0-2636c979075f\") "
Oct 30 22:28:25.619310 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:25.619240    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f137aa3-0e26-4526-92b0-2636c979075f-istio-data" pod "9f137aa3-0e26-4526-92b0-2636c979075f" (UID: "9f137aa3-0e26-4526-92b0-2636c979075f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:28:25.619574 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:25.619515    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f137aa3-0e26-4526-92b0-2636c979075f-workload-socket" pod "9f137aa3-0e26-4526-92b0-2636c979075f" (UID: "9f137aa3-0e26-4526-92b0-2636c979075f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:28:25.619794 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:25.619699    5061 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f137aa3-0e26-4526-92b0-2636c979075f-istiod-ca-cert\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:25.619794 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:25.619723    5061 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f137aa3-0e26-4526-92b0-2636c979075f-istio-data\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:25.619794 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:25.619737    5061 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f137aa3-0e26-4526-92b0-2636c979075f-workload-socket\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:25.620032 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:25.619829    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f137aa3-0e26-4526-92b0-2636c979075f-credential-socket" pod "9f137aa3-0e26-4526-92b0-2636c979075f" (UID: "9f137aa3-0e26-4526-92b0-2636c979075f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:28:25.620103 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:25.620074    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f137aa3-0e26-4526-92b0-2636c979075f-workload-certs" pod "9f137aa3-0e26-4526-92b0-2636c979075f" (UID: "9f137aa3-0e26-4526-92b0-2636c979075f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:28:25.621260 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:25.621235    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f137aa3-0e26-4526-92b0-2636c979075f-kube-api-access-jsxp6" pod "9f137aa3-0e26-4526-92b0-2636c979075f" (UID: "9f137aa3-0e26-4526-92b0-2636c979075f"). InnerVolumeSpecName "kube-api-access-jsxp6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:28:25.621394 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:25.621378    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9f137aa3-0e26-4526-92b0-2636c979075f-istio-podinfo" pod "9f137aa3-0e26-4526-92b0-2636c979075f" (UID: "9f137aa3-0e26-4526-92b0-2636c979075f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:28:25.621457 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:25.621379    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f137aa3-0e26-4526-92b0-2636c979075f-istio-token" pod "9f137aa3-0e26-4526-92b0-2636c979075f" (UID: "9f137aa3-0e26-4526-92b0-2636c979075f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:28:25.621511 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:25.621466    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f137aa3-0e26-4526-92b0-2636c979075f-istio-envoy" pod "9f137aa3-0e26-4526-92b0-2636c979075f" (UID: "9f137aa3-0e26-4526-92b0-2636c979075f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:28:25.720754 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:25.720718    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jsxp6\" (UniqueName: \"kubernetes.io/projected/9f137aa3-0e26-4526-92b0-2636c979075f-kube-api-access-jsxp6\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:25.720754 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:25.720745    5061 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f137aa3-0e26-4526-92b0-2636c979075f-istio-podinfo\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:25.720754 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:25.720758    5061 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f137aa3-0e26-4526-92b0-2636c979075f-istio-envoy\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:25.721175 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:25.720771    5061 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f137aa3-0e26-4526-92b0-2636c979075f-credential-socket\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:25.721175 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:25.720783    5061 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f137aa3-0e26-4526-92b0-2636c979075f-istio-token\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:25.721175 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:25.720794    5061 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f137aa3-0e26-4526-92b0-2636c979075f-workload-certs\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:25.786314 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:25.786277    5061 generic.go:356] "Generic (PLEG): container finished" podID="9f137aa3-0e26-4526-92b0-2636c979075f" containerID="c85d056c85a8b6e789e406e2e1db65cd780b651b9239857a721b511f348c1b76" exitCode=0
Oct 30 22:28:25.786508 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:25.786329    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-shwf8"
Oct 30 22:28:25.786508 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:25.786374    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-shwf8" event={"ID":"9f137aa3-0e26-4526-92b0-2636c979075f","Type":"ContainerDied","Data":"c85d056c85a8b6e789e406e2e1db65cd780b651b9239857a721b511f348c1b76"}
Oct 30 22:28:25.786508 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:25.786426    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-shwf8" event={"ID":"9f137aa3-0e26-4526-92b0-2636c979075f","Type":"ContainerDied","Data":"829e67ec674c7f44931ff9675f91091236f1f795da8f5999295d1fb04bcb1ac6"}
Oct 30 22:28:25.786508 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:25.786447    5061 scope.go:122] "RemoveContainer" containerID="c85d056c85a8b6e789e406e2e1db65cd780b651b9239857a721b511f348c1b76"
Oct 30 22:28:25.797092 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:25.797076    5061 scope.go:122] "RemoveContainer" containerID="c85d056c85a8b6e789e406e2e1db65cd780b651b9239857a721b511f348c1b76"
Oct 30 22:28:25.797363 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:28:25.797346    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c85d056c85a8b6e789e406e2e1db65cd780b651b9239857a721b511f348c1b76\": container with ID starting with c85d056c85a8b6e789e406e2e1db65cd780b651b9239857a721b511f348c1b76 not found: ID does not exist" containerID="c85d056c85a8b6e789e406e2e1db65cd780b651b9239857a721b511f348c1b76"
Oct 30 22:28:25.797423 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:25.797369    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c85d056c85a8b6e789e406e2e1db65cd780b651b9239857a721b511f348c1b76"} err="failed to get container status \"c85d056c85a8b6e789e406e2e1db65cd780b651b9239857a721b511f348c1b76\": rpc error: code = NotFound desc = could not find container \"c85d056c85a8b6e789e406e2e1db65cd780b651b9239857a721b511f348c1b76\": container with ID starting with c85d056c85a8b6e789e406e2e1db65cd780b651b9239857a721b511f348c1b76 not found: ID does not exist"
Oct 30 22:28:25.946095 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:25.946024    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-shwf8"]
Oct 30 22:28:25.952331 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:25.952306    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-shwf8"]
Oct 30 22:28:26.527628 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:26.527594    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f137aa3-0e26-4526-92b0-2636c979075f" path="/var/lib/kubelet/pods/9f137aa3-0e26-4526-92b0-2636c979075f/volumes"
Oct 30 22:28:29.162980 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:29.162947    5061 scope.go:122] "RemoveContainer" containerID="f9029a35a96288f6f645c0720c4912faf866e95e51535c6c1cb5fb3c0834a121"
Oct 30 22:28:41.072149 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:41.072109    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f028c46f-tpj6q"]
Oct 30 22:28:41.072804 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:41.072781    5061 cpu_manager.go:414] "containerMap: removing container" podUID="4646779e-fc85-4860-adcf-9855d2d26fb1" containerName="maas-api"
Oct 30 22:28:41.072916 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:41.072803    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4646779e-fc85-4860-adcf-9855d2d26fb1" containerName="maas-api"
Oct 30 22:28:41.072916 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:41.072829    5061 cpu_manager.go:414] "containerMap: removing container" podUID="6977b446-b5f9-4f27-8d41-a4942379c9b2" containerName="payload-processing"
Oct 30 22:28:41.072916 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:41.072835    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6977b446-b5f9-4f27-8d41-a4942379c9b2" containerName="payload-processing"
Oct 30 22:28:41.072916 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:41.072842    5061 cpu_manager.go:414] "containerMap: removing container" podUID="9fc81a52-bb17-4dc5-9de2-62ae34ee7917" containerName="revoke-keys"
Oct 30 22:28:41.072916 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:41.072847    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fc81a52-bb17-4dc5-9de2-62ae34ee7917" containerName="revoke-keys"
Oct 30 22:28:41.072916 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:41.072853    5061 cpu_manager.go:414] "containerMap: removing container" podUID="44d23433-b77d-49d4-a5a3-a969f2d0507d" containerName="revoke-keys"
Oct 30 22:28:41.072916 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:41.072858    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44d23433-b77d-49d4-a5a3-a969f2d0507d" containerName="revoke-keys"
Oct 30 22:28:41.072916 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:41.072864    5061 cpu_manager.go:414] "containerMap: removing container" podUID="9f137aa3-0e26-4526-92b0-2636c979075f" containerName="istio-proxy"
Oct 30 22:28:41.072916 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:41.072869    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f137aa3-0e26-4526-92b0-2636c979075f" containerName="istio-proxy"
Oct 30 22:28:41.073381 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:41.072971    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="4646779e-fc85-4860-adcf-9855d2d26fb1" containerName="maas-api"
Oct 30 22:28:41.073381 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:41.072981    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="6977b446-b5f9-4f27-8d41-a4942379c9b2" containerName="payload-processing"
Oct 30 22:28:41.073381 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:41.072986    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fc81a52-bb17-4dc5-9de2-62ae34ee7917" containerName="revoke-keys"
Oct 30 22:28:41.073381 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:41.072991    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="44d23433-b77d-49d4-a5a3-a969f2d0507d" containerName="revoke-keys"
Oct 30 22:28:41.073381 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:41.072995    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f137aa3-0e26-4526-92b0-2636c979075f" containerName="istio-proxy"
Oct 30 22:28:41.078459 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:41.078437    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f028c46f-tpj6q"
Oct 30 22:28:41.135273 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:41.135246    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gdmfg\""
Oct 30 22:28:41.169981 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:41.169946    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f4cd395-f0a0-49dc-9deb-4eb3b5d2c67f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f028c46f-tpj6q\" (UID: \"4f4cd395-f0a0-49dc-9deb-4eb3b5d2c67f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f028c46f-tpj6q"
Oct 30 22:28:41.170140 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:41.170032    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9wf8\" (UniqueName: \"kubernetes.io/projected/4f4cd395-f0a0-49dc-9deb-4eb3b5d2c67f-kube-api-access-j9wf8\") pod \"maas-api-revoke-sub-f028c46f-tpj6q\" (UID: \"4f4cd395-f0a0-49dc-9deb-4eb3b5d2c67f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f028c46f-tpj6q"
Oct 30 22:28:41.170140 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:41.170097    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f4cd395-f0a0-49dc-9deb-4eb3b5d2c67f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f028c46f-tpj6q\" (UID: \"4f4cd395-f0a0-49dc-9deb-4eb3b5d2c67f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f028c46f-tpj6q"
Oct 30 22:28:41.182755 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:41.182729    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:28:41.182920 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:41.182903    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:28:41.185652 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:41.185631    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f028c46f-tpj6q"]
Oct 30 22:28:41.231162 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:41.231133    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:28:41.271067 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:41.271037    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f4cd395-f0a0-49dc-9deb-4eb3b5d2c67f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f028c46f-tpj6q\" (UID: \"4f4cd395-f0a0-49dc-9deb-4eb3b5d2c67f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f028c46f-tpj6q"
Oct 30 22:28:41.271182 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:41.271097    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9wf8\" (UniqueName: \"kubernetes.io/projected/4f4cd395-f0a0-49dc-9deb-4eb3b5d2c67f-kube-api-access-j9wf8\") pod \"maas-api-revoke-sub-f028c46f-tpj6q\" (UID: \"4f4cd395-f0a0-49dc-9deb-4eb3b5d2c67f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f028c46f-tpj6q"
Oct 30 22:28:41.271182 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:41.271124    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f4cd395-f0a0-49dc-9deb-4eb3b5d2c67f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f028c46f-tpj6q\" (UID: \"4f4cd395-f0a0-49dc-9deb-4eb3b5d2c67f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f028c46f-tpj6q"
Oct 30 22:28:41.271740 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:41.271710    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f4cd395-f0a0-49dc-9deb-4eb3b5d2c67f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f028c46f-tpj6q\" (UID: \"4f4cd395-f0a0-49dc-9deb-4eb3b5d2c67f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f028c46f-tpj6q"
Oct 30 22:28:41.273396 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:41.273374    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f4cd395-f0a0-49dc-9deb-4eb3b5d2c67f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f028c46f-tpj6q\" (UID: \"4f4cd395-f0a0-49dc-9deb-4eb3b5d2c67f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f028c46f-tpj6q"
Oct 30 22:28:41.326368 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:41.326287    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9wf8\" (UniqueName: \"kubernetes.io/projected/4f4cd395-f0a0-49dc-9deb-4eb3b5d2c67f-kube-api-access-j9wf8\") pod \"maas-api-revoke-sub-f028c46f-tpj6q\" (UID: \"4f4cd395-f0a0-49dc-9deb-4eb3b5d2c67f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f028c46f-tpj6q"
Oct 30 22:28:41.391962 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:41.391937    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f028c46f-tpj6q"
Oct 30 22:28:41.771466 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:41.771249    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f028c46f-tpj6q"]
Oct 30 22:28:41.774981 ip-10-0-130-60 kubenswrapper[5061]: W1030 22:28:41.774951    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f4cd395_f0a0_49dc_9deb_4eb3b5d2c67f.slice/crio-c544bd63a9512f05b38e0530d9fbf4349560edce9214cdbf66ebb2cc43cf7eed WatchSource:0}: Error finding container c544bd63a9512f05b38e0530d9fbf4349560edce9214cdbf66ebb2cc43cf7eed: Status 404 returned error can't find the container with id c544bd63a9512f05b38e0530d9fbf4349560edce9214cdbf66ebb2cc43cf7eed
Oct 30 22:28:41.897487 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:41.897429    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f028c46f-tpj6q" event={"ID":"4f4cd395-f0a0-49dc-9deb-4eb3b5d2c67f","Type":"ContainerStarted","Data":"c544bd63a9512f05b38e0530d9fbf4349560edce9214cdbf66ebb2cc43cf7eed"}
Oct 30 22:28:42.904922 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:42.904828    5061 generic.go:356] "Generic (PLEG): container finished" podID="4f4cd395-f0a0-49dc-9deb-4eb3b5d2c67f" containerID="c930146fb09d07e726e7cfc57e0c53222dcb6761002bc6440af5104f6cfab684" exitCode=0
Oct 30 22:28:42.904922 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:42.904872    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f028c46f-tpj6q" event={"ID":"4f4cd395-f0a0-49dc-9deb-4eb3b5d2c67f","Type":"ContainerDied","Data":"c930146fb09d07e726e7cfc57e0c53222dcb6761002bc6440af5104f6cfab684"}
Oct 30 22:28:44.062189 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:44.062166    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f028c46f-tpj6q"
Oct 30 22:28:44.102108 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:44.102083    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f4cd395-f0a0-49dc-9deb-4eb3b5d2c67f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f4cd395-f0a0-49dc-9deb-4eb3b5d2c67f-maas-api-openshift-ca\") pod \"4f4cd395-f0a0-49dc-9deb-4eb3b5d2c67f\" (UID: \"4f4cd395-f0a0-49dc-9deb-4eb3b5d2c67f\") "
Oct 30 22:28:44.102227 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:44.102120    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f4cd395-f0a0-49dc-9deb-4eb3b5d2c67f-kube-api-access-j9wf8\" (UniqueName: \"kubernetes.io/projected/4f4cd395-f0a0-49dc-9deb-4eb3b5d2c67f-kube-api-access-j9wf8\") pod \"4f4cd395-f0a0-49dc-9deb-4eb3b5d2c67f\" (UID: \"4f4cd395-f0a0-49dc-9deb-4eb3b5d2c67f\") "
Oct 30 22:28:44.102227 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:44.102181    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f4cd395-f0a0-49dc-9deb-4eb3b5d2c67f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f4cd395-f0a0-49dc-9deb-4eb3b5d2c67f-maas-api-xks-ca\") pod \"4f4cd395-f0a0-49dc-9deb-4eb3b5d2c67f\" (UID: \"4f4cd395-f0a0-49dc-9deb-4eb3b5d2c67f\") "
Oct 30 22:28:44.102449 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:44.102425    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f4cd395-f0a0-49dc-9deb-4eb3b5d2c67f-maas-api-openshift-ca" pod "4f4cd395-f0a0-49dc-9deb-4eb3b5d2c67f" (UID: "4f4cd395-f0a0-49dc-9deb-4eb3b5d2c67f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:28:44.102653 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:44.102638    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f4cd395-f0a0-49dc-9deb-4eb3b5d2c67f-maas-api-openshift-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:44.104362 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:44.104341    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f4cd395-f0a0-49dc-9deb-4eb3b5d2c67f-kube-api-access-j9wf8" pod "4f4cd395-f0a0-49dc-9deb-4eb3b5d2c67f" (UID: "4f4cd395-f0a0-49dc-9deb-4eb3b5d2c67f"). InnerVolumeSpecName "kube-api-access-j9wf8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:28:44.104429 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:44.104344    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f4cd395-f0a0-49dc-9deb-4eb3b5d2c67f-maas-api-xks-ca" pod "4f4cd395-f0a0-49dc-9deb-4eb3b5d2c67f" (UID: "4f4cd395-f0a0-49dc-9deb-4eb3b5d2c67f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:28:44.203351 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:44.203275    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f4cd395-f0a0-49dc-9deb-4eb3b5d2c67f-maas-api-xks-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:44.203351 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:44.203303    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9wf8\" (UniqueName: \"kubernetes.io/projected/4f4cd395-f0a0-49dc-9deb-4eb3b5d2c67f-kube-api-access-j9wf8\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:44.919772 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:44.919736    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f028c46f-tpj6q"
Oct 30 22:28:44.919772 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:44.919747    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f028c46f-tpj6q" event={"ID":"4f4cd395-f0a0-49dc-9deb-4eb3b5d2c67f","Type":"ContainerDied","Data":"c544bd63a9512f05b38e0530d9fbf4349560edce9214cdbf66ebb2cc43cf7eed"}
Oct 30 22:28:44.920031 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:44.919797    5061 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c544bd63a9512f05b38e0530d9fbf4349560edce9214cdbf66ebb2cc43cf7eed"
Oct 30 22:28:52.985884 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:52.985852    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-3aa85e/test-model-3aa85e-kserve-7dc5f8f957-99956"]
Oct 30 22:28:53.930916 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:53.930880    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89af4e0a-blf2f"]
Oct 30 22:28:53.931371 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:53.931358    5061 cpu_manager.go:414] "containerMap: removing container" podUID="4f4cd395-f0a0-49dc-9deb-4eb3b5d2c67f" containerName="revoke-keys"
Oct 30 22:28:53.931438 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:53.931372    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f4cd395-f0a0-49dc-9deb-4eb3b5d2c67f" containerName="revoke-keys"
Oct 30 22:28:53.931491 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:53.931457    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f4cd395-f0a0-49dc-9deb-4eb3b5d2c67f" containerName="revoke-keys"
Oct 30 22:28:53.935294 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:53.935272    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89af4e0a-blf2f"
Oct 30 22:28:53.984902 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:53.984879    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:28:53.997564 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:53.997510    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npz4j\" (UniqueName: \"kubernetes.io/projected/577603c3-9fdf-42c5-be4f-ac2533a9c4f0-kube-api-access-npz4j\") pod \"maas-api-revoke-sub-89af4e0a-blf2f\" (UID: \"577603c3-9fdf-42c5-be4f-ac2533a9c4f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89af4e0a-blf2f"
Oct 30 22:28:53.997868 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:53.997653    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/577603c3-9fdf-42c5-be4f-ac2533a9c4f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89af4e0a-blf2f\" (UID: \"577603c3-9fdf-42c5-be4f-ac2533a9c4f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89af4e0a-blf2f"
Oct 30 22:28:53.997868 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:53.997814    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/577603c3-9fdf-42c5-be4f-ac2533a9c4f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89af4e0a-blf2f\" (UID: \"577603c3-9fdf-42c5-be4f-ac2533a9c4f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89af4e0a-blf2f"
Oct 30 22:28:54.031769 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:54.031732    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:28:54.031974 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:54.031953    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:28:54.032021 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:54.031964    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gdmfg\""
Oct 30 22:28:54.034979 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:54.034958    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89af4e0a-blf2f"]
Oct 30 22:28:54.085030 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:54.085006    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9563d02b-vbxjb"]
Oct 30 22:28:54.087282 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:54.087257    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9563d02b-vbxjb"]
Oct 30 22:28:54.098886 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:54.098862    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/577603c3-9fdf-42c5-be4f-ac2533a9c4f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89af4e0a-blf2f\" (UID: \"577603c3-9fdf-42c5-be4f-ac2533a9c4f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89af4e0a-blf2f"
Oct 30 22:28:54.098996 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:54.098910    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npz4j\" (UniqueName: \"kubernetes.io/projected/577603c3-9fdf-42c5-be4f-ac2533a9c4f0-kube-api-access-npz4j\") pod \"maas-api-revoke-sub-89af4e0a-blf2f\" (UID: \"577603c3-9fdf-42c5-be4f-ac2533a9c4f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89af4e0a-blf2f"
Oct 30 22:28:54.098996 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:54.098954    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/577603c3-9fdf-42c5-be4f-ac2533a9c4f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89af4e0a-blf2f\" (UID: \"577603c3-9fdf-42c5-be4f-ac2533a9c4f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89af4e0a-blf2f"
Oct 30 22:28:54.099608 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:54.099589    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/577603c3-9fdf-42c5-be4f-ac2533a9c4f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89af4e0a-blf2f\" (UID: \"577603c3-9fdf-42c5-be4f-ac2533a9c4f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89af4e0a-blf2f"
Oct 30 22:28:54.101235 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:54.101214    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/577603c3-9fdf-42c5-be4f-ac2533a9c4f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89af4e0a-blf2f\" (UID: \"577603c3-9fdf-42c5-be4f-ac2533a9c4f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89af4e0a-blf2f"
Oct 30 22:28:54.155219 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:54.155195    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npz4j\" (UniqueName: \"kubernetes.io/projected/577603c3-9fdf-42c5-be4f-ac2533a9c4f0-kube-api-access-npz4j\") pod \"maas-api-revoke-sub-89af4e0a-blf2f\" (UID: \"577603c3-9fdf-42c5-be4f-ac2533a9c4f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89af4e0a-blf2f"
Oct 30 22:28:54.248129 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:54.248069    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89af4e0a-blf2f"
Oct 30 22:28:54.528305 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:54.528236    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44457901-5c4b-48da-84ee-e5a5fca8b14e" path="/var/lib/kubelet/pods/44457901-5c4b-48da-84ee-e5a5fca8b14e/volumes"
Oct 30 22:28:54.615765 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:54.615733    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89af4e0a-blf2f"]
Oct 30 22:28:54.616312 ip-10-0-130-60 kubenswrapper[5061]: W1030 22:28:54.616289    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod577603c3_9fdf_42c5_be4f_ac2533a9c4f0.slice/crio-49f4bdccb9af0f2916dd4ed805f10bb1ca17f8c8abb6d33332ffc07bb4b9cbb5 WatchSource:0}: Error finding container 49f4bdccb9af0f2916dd4ed805f10bb1ca17f8c8abb6d33332ffc07bb4b9cbb5: Status 404 returned error can't find the container with id 49f4bdccb9af0f2916dd4ed805f10bb1ca17f8c8abb6d33332ffc07bb4b9cbb5
Oct 30 22:28:54.670719 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:28:54.670687    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 22:28:54.983093 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:54.983052    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89af4e0a-blf2f" event={"ID":"577603c3-9fdf-42c5-be4f-ac2533a9c4f0","Type":"ContainerStarted","Data":"ba65bc0bb1e8841ec2f2c0b1550de5ae975998183e7a5a6ec4028aae2167ada1"}
Oct 30 22:28:54.983266 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:54.983104    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89af4e0a-blf2f" event={"ID":"577603c3-9fdf-42c5-be4f-ac2533a9c4f0","Type":"ContainerStarted","Data":"49f4bdccb9af0f2916dd4ed805f10bb1ca17f8c8abb6d33332ffc07bb4b9cbb5"}
Oct 30 22:28:55.050032 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:55.049998    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c806f79-btkzb"]
Oct 30 22:28:55.052671 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:55.052636    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c806f79-btkzb"]
Oct 30 22:28:55.094949 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:55.094873    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89af4e0a-blf2f" podStartSLOduration=2.094860758 podStartE2EDuration="2.094860758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:28:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:28:55.093688918 +0000 UTC m=+2788.706458699" watchObservedRunningTime="2026-10-30 22:28:55.094860758 +0000 UTC m=+2788.707630515"
Oct 30 22:28:55.991800 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:55.991764    5061 generic.go:356] "Generic (PLEG): container finished" podID="577603c3-9fdf-42c5-be4f-ac2533a9c4f0" containerID="ba65bc0bb1e8841ec2f2c0b1550de5ae975998183e7a5a6ec4028aae2167ada1" exitCode=0
Oct 30 22:28:55.991976 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:55.991822    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89af4e0a-blf2f" event={"ID":"577603c3-9fdf-42c5-be4f-ac2533a9c4f0","Type":"ContainerDied","Data":"ba65bc0bb1e8841ec2f2c0b1550de5ae975998183e7a5a6ec4028aae2167ada1"}
Oct 30 22:28:56.529523 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:56.529487    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00b6569b-f05f-4260-b9ea-a0dbb6f1a926" path="/var/lib/kubelet/pods/00b6569b-f05f-4260-b9ea-a0dbb6f1a926/volumes"
Oct 30 22:28:57.146247 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:57.146216    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89af4e0a-blf2f"
Oct 30 22:28:57.232324 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:57.232292    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/577603c3-9fdf-42c5-be4f-ac2533a9c4f0-kube-api-access-npz4j\" (UniqueName: \"kubernetes.io/projected/577603c3-9fdf-42c5-be4f-ac2533a9c4f0-kube-api-access-npz4j\") pod \"577603c3-9fdf-42c5-be4f-ac2533a9c4f0\" (UID: \"577603c3-9fdf-42c5-be4f-ac2533a9c4f0\") "
Oct 30 22:28:57.232499 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:57.232352    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/577603c3-9fdf-42c5-be4f-ac2533a9c4f0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/577603c3-9fdf-42c5-be4f-ac2533a9c4f0-maas-api-xks-ca\") pod \"577603c3-9fdf-42c5-be4f-ac2533a9c4f0\" (UID: \"577603c3-9fdf-42c5-be4f-ac2533a9c4f0\") "
Oct 30 22:28:57.232499 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:57.232387    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/577603c3-9fdf-42c5-be4f-ac2533a9c4f0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/577603c3-9fdf-42c5-be4f-ac2533a9c4f0-maas-api-openshift-ca\") pod \"577603c3-9fdf-42c5-be4f-ac2533a9c4f0\" (UID: \"577603c3-9fdf-42c5-be4f-ac2533a9c4f0\") "
Oct 30 22:28:57.232830 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:57.232803    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/577603c3-9fdf-42c5-be4f-ac2533a9c4f0-maas-api-openshift-ca" pod "577603c3-9fdf-42c5-be4f-ac2533a9c4f0" (UID: "577603c3-9fdf-42c5-be4f-ac2533a9c4f0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:28:57.234622 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:57.234588    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/577603c3-9fdf-42c5-be4f-ac2533a9c4f0-kube-api-access-npz4j" pod "577603c3-9fdf-42c5-be4f-ac2533a9c4f0" (UID: "577603c3-9fdf-42c5-be4f-ac2533a9c4f0"). InnerVolumeSpecName "kube-api-access-npz4j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:28:57.234736 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:57.234695    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/577603c3-9fdf-42c5-be4f-ac2533a9c4f0-maas-api-xks-ca" pod "577603c3-9fdf-42c5-be4f-ac2533a9c4f0" (UID: "577603c3-9fdf-42c5-be4f-ac2533a9c4f0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:28:57.333937 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:57.333905    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/577603c3-9fdf-42c5-be4f-ac2533a9c4f0-maas-api-xks-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:57.333937 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:57.333934    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/577603c3-9fdf-42c5-be4f-ac2533a9c4f0-maas-api-openshift-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:57.333937 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:57.333945    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npz4j\" (UniqueName: \"kubernetes.io/projected/577603c3-9fdf-42c5-be4f-ac2533a9c4f0-kube-api-access-npz4j\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:58.005445 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:58.005407    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89af4e0a-blf2f"
Oct 30 22:28:58.005445 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:58.005436    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89af4e0a-blf2f" event={"ID":"577603c3-9fdf-42c5-be4f-ac2533a9c4f0","Type":"ContainerDied","Data":"49f4bdccb9af0f2916dd4ed805f10bb1ca17f8c8abb6d33332ffc07bb4b9cbb5"}
Oct 30 22:28:58.005936 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:28:58.005479    5061 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="49f4bdccb9af0f2916dd4ed805f10bb1ca17f8c8abb6d33332ffc07bb4b9cbb5"
Oct 30 22:29:08.005684 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:08.005627    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-3aa85e/test-model-3aa85e-kserve-7dc5f8f957-99956" podUID="18626f54-af52-42d0-a879-be301c98a500" containerName="main" containerID="cri-o://fbd2aa98fdee13aad0e19acd34b0994c242bba89148a091fbfaa1c0c75b93fa2" gracePeriod=45
Oct 30 22:29:08.287477 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:08.287450    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-3aa85e/test-model-3aa85e-kserve-7dc5f8f957-99956"
Oct 30 22:29:08.448006 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:08.447972    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18626f54-af52-42d0-a879-be301c98a500-tls-certs\" (UniqueName: \"kubernetes.io/secret/18626f54-af52-42d0-a879-be301c98a500-tls-certs\") pod \"18626f54-af52-42d0-a879-be301c98a500\" (UID: \"18626f54-af52-42d0-a879-be301c98a500\") "
Oct 30 22:29:08.448200 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:08.448118    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18626f54-af52-42d0-a879-be301c98a500-kube-api-access-brg5m\" (UniqueName: \"kubernetes.io/projected/18626f54-af52-42d0-a879-be301c98a500-kube-api-access-brg5m\") pod \"18626f54-af52-42d0-a879-be301c98a500\" (UID: \"18626f54-af52-42d0-a879-be301c98a500\") "
Oct 30 22:29:08.448200 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:08.448191    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18626f54-af52-42d0-a879-be301c98a500-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18626f54-af52-42d0-a879-be301c98a500-tmp-dir\") pod \"18626f54-af52-42d0-a879-be301c98a500\" (UID: \"18626f54-af52-42d0-a879-be301c98a500\") "
Oct 30 22:29:08.448323 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:08.448237    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18626f54-af52-42d0-a879-be301c98a500-dshm\" (UniqueName: \"kubernetes.io/empty-dir/18626f54-af52-42d0-a879-be301c98a500-dshm\") pod \"18626f54-af52-42d0-a879-be301c98a500\" (UID: \"18626f54-af52-42d0-a879-be301c98a500\") "
Oct 30 22:29:08.448323 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:08.448276    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18626f54-af52-42d0-a879-be301c98a500-home\" (UniqueName: \"kubernetes.io/empty-dir/18626f54-af52-42d0-a879-be301c98a500-home\") pod \"18626f54-af52-42d0-a879-be301c98a500\" (UID: \"18626f54-af52-42d0-a879-be301c98a500\") "
Oct 30 22:29:08.448442 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:08.448328    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18626f54-af52-42d0-a879-be301c98a500-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18626f54-af52-42d0-a879-be301c98a500-model-cache\") pod \"18626f54-af52-42d0-a879-be301c98a500\" (UID: \"18626f54-af52-42d0-a879-be301c98a500\") "
Oct 30 22:29:08.448498 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:08.448442    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18626f54-af52-42d0-a879-be301c98a500-tmp-dir" pod "18626f54-af52-42d0-a879-be301c98a500" (UID: "18626f54-af52-42d0-a879-be301c98a500"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:08.448606 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:08.448561    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18626f54-af52-42d0-a879-be301c98a500-home" pod "18626f54-af52-42d0-a879-be301c98a500" (UID: "18626f54-af52-42d0-a879-be301c98a500"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:08.448676 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:08.448640    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18626f54-af52-42d0-a879-be301c98a500-model-cache" pod "18626f54-af52-42d0-a879-be301c98a500" (UID: "18626f54-af52-42d0-a879-be301c98a500"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:08.448806 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:08.448786    5061 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18626f54-af52-42d0-a879-be301c98a500-tmp-dir\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:08.448870 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:08.448810    5061 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18626f54-af52-42d0-a879-be301c98a500-home\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:08.448870 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:08.448826    5061 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18626f54-af52-42d0-a879-be301c98a500-model-cache\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:08.450323 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:08.450289    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18626f54-af52-42d0-a879-be301c98a500-dshm" pod "18626f54-af52-42d0-a879-be301c98a500" (UID: "18626f54-af52-42d0-a879-be301c98a500"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:08.450454 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:08.450333    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18626f54-af52-42d0-a879-be301c98a500-kube-api-access-brg5m" pod "18626f54-af52-42d0-a879-be301c98a500" (UID: "18626f54-af52-42d0-a879-be301c98a500"). InnerVolumeSpecName "kube-api-access-brg5m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:29:08.450454 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:08.450403    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18626f54-af52-42d0-a879-be301c98a500-tls-certs" pod "18626f54-af52-42d0-a879-be301c98a500" (UID: "18626f54-af52-42d0-a879-be301c98a500"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:29:08.550054 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:08.550027    5061 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18626f54-af52-42d0-a879-be301c98a500-dshm\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:08.550054 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:08.550055    5061 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18626f54-af52-42d0-a879-be301c98a500-tls-certs\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:08.550211 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:08.550065    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brg5m\" (UniqueName: \"kubernetes.io/projected/18626f54-af52-42d0-a879-be301c98a500-kube-api-access-brg5m\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:09.078955 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:09.078918    5061 generic.go:356] "Generic (PLEG): container finished" podID="18626f54-af52-42d0-a879-be301c98a500" containerID="fbd2aa98fdee13aad0e19acd34b0994c242bba89148a091fbfaa1c0c75b93fa2" exitCode=0
Oct 30 22:29:09.079469 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:09.078971    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-3aa85e/test-model-3aa85e-kserve-7dc5f8f957-99956"
Oct 30 22:29:09.079469 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:09.079001    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-3aa85e/test-model-3aa85e-kserve-7dc5f8f957-99956" event={"ID":"18626f54-af52-42d0-a879-be301c98a500","Type":"ContainerDied","Data":"fbd2aa98fdee13aad0e19acd34b0994c242bba89148a091fbfaa1c0c75b93fa2"}
Oct 30 22:29:09.079469 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:09.079058    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-3aa85e/test-model-3aa85e-kserve-7dc5f8f957-99956" event={"ID":"18626f54-af52-42d0-a879-be301c98a500","Type":"ContainerDied","Data":"dfd587781b01649c2bfba9cda3a091049ef5800723e2f60e22241d78d3afe35e"}
Oct 30 22:29:09.079469 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:09.079082    5061 scope.go:122] "RemoveContainer" containerID="fbd2aa98fdee13aad0e19acd34b0994c242bba89148a091fbfaa1c0c75b93fa2"
Oct 30 22:29:09.089350 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:09.089325    5061 scope.go:122] "RemoveContainer" containerID="fbd2aa98fdee13aad0e19acd34b0994c242bba89148a091fbfaa1c0c75b93fa2"
Oct 30 22:29:09.089685 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:29:09.089666    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbd2aa98fdee13aad0e19acd34b0994c242bba89148a091fbfaa1c0c75b93fa2\": container with ID starting with fbd2aa98fdee13aad0e19acd34b0994c242bba89148a091fbfaa1c0c75b93fa2 not found: ID does not exist" containerID="fbd2aa98fdee13aad0e19acd34b0994c242bba89148a091fbfaa1c0c75b93fa2"
Oct 30 22:29:09.089769 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:09.089696    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbd2aa98fdee13aad0e19acd34b0994c242bba89148a091fbfaa1c0c75b93fa2"} err="failed to get container status \"fbd2aa98fdee13aad0e19acd34b0994c242bba89148a091fbfaa1c0c75b93fa2\": rpc error: code = NotFound desc = could not find container \"fbd2aa98fdee13aad0e19acd34b0994c242bba89148a091fbfaa1c0c75b93fa2\": container with ID starting with fbd2aa98fdee13aad0e19acd34b0994c242bba89148a091fbfaa1c0c75b93fa2 not found: ID does not exist"
Oct 30 22:29:09.234490 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:09.234453    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-3aa85e/test-model-3aa85e-kserve-7dc5f8f957-99956"]
Oct 30 22:29:09.281474 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:09.281432    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-3aa85e/test-model-3aa85e-kserve-7dc5f8f957-99956"]
Oct 30 22:29:10.527810 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:10.527775    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18626f54-af52-42d0-a879-be301c98a500" path="/var/lib/kubelet/pods/18626f54-af52-42d0-a879-be301c98a500/volumes"
Oct 30 22:29:14.638717 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:14.638678    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-84a3cf/test-model-84a3cf-kserve-57554f46fc-j6rcb"]
Oct 30 22:29:15.192896 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:15.192857    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3aa85e-vt4l4"]
Oct 30 22:29:15.193397 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:15.193383    5061 cpu_manager.go:414] "containerMap: removing container" podUID="577603c3-9fdf-42c5-be4f-ac2533a9c4f0" containerName="revoke-keys"
Oct 30 22:29:15.193443 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:15.193397    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="577603c3-9fdf-42c5-be4f-ac2533a9c4f0" containerName="revoke-keys"
Oct 30 22:29:15.193443 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:15.193425    5061 cpu_manager.go:414] "containerMap: removing container" podUID="18626f54-af52-42d0-a879-be301c98a500" containerName="main"
Oct 30 22:29:15.193443 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:15.193432    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18626f54-af52-42d0-a879-be301c98a500" containerName="main"
Oct 30 22:29:15.193699 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:15.193522    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="18626f54-af52-42d0-a879-be301c98a500" containerName="main"
Oct 30 22:29:15.193699 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:15.193558    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="577603c3-9fdf-42c5-be4f-ac2533a9c4f0" containerName="revoke-keys"
Oct 30 22:29:15.198294 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:15.198272    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3aa85e-vt4l4"
Oct 30 22:29:15.222816 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:15.222789    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7h86\" (UniqueName: \"kubernetes.io/projected/2999b63a-282f-4e12-9ca8-4004dc67a62f-kube-api-access-c7h86\") pod \"maas-api-revoke-keys-e2e-inf-a-3aa85e-vt4l4\" (UID: \"2999b63a-282f-4e12-9ca8-4004dc67a62f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3aa85e-vt4l4"
Oct 30 22:29:15.222947 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:15.222832    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2999b63a-282f-4e12-9ca8-4004dc67a62f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-3aa85e-vt4l4\" (UID: \"2999b63a-282f-4e12-9ca8-4004dc67a62f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3aa85e-vt4l4"
Oct 30 22:29:15.222947 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:15.222857    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2999b63a-282f-4e12-9ca8-4004dc67a62f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-3aa85e-vt4l4\" (UID: \"2999b63a-282f-4e12-9ca8-4004dc67a62f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3aa85e-vt4l4"
Oct 30 22:29:15.247997 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:15.247962    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:29:15.247997 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:15.247969    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:29:15.248174 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:15.248010    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:29:15.294789 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:15.294752    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gdmfg\""
Oct 30 22:29:15.297580 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:15.297552    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3aa85e-vt4l4"]
Oct 30 22:29:15.324129 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:15.324102    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2999b63a-282f-4e12-9ca8-4004dc67a62f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-3aa85e-vt4l4\" (UID: \"2999b63a-282f-4e12-9ca8-4004dc67a62f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3aa85e-vt4l4"
Oct 30 22:29:15.324258 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:15.324145    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2999b63a-282f-4e12-9ca8-4004dc67a62f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-3aa85e-vt4l4\" (UID: \"2999b63a-282f-4e12-9ca8-4004dc67a62f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3aa85e-vt4l4"
Oct 30 22:29:15.324258 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:15.324250    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7h86\" (UniqueName: \"kubernetes.io/projected/2999b63a-282f-4e12-9ca8-4004dc67a62f-kube-api-access-c7h86\") pod \"maas-api-revoke-keys-e2e-inf-a-3aa85e-vt4l4\" (UID: \"2999b63a-282f-4e12-9ca8-4004dc67a62f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3aa85e-vt4l4"
Oct 30 22:29:15.324823 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:15.324801    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2999b63a-282f-4e12-9ca8-4004dc67a62f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-3aa85e-vt4l4\" (UID: \"2999b63a-282f-4e12-9ca8-4004dc67a62f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3aa85e-vt4l4"
Oct 30 22:29:15.326537 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:15.326505    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2999b63a-282f-4e12-9ca8-4004dc67a62f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-3aa85e-vt4l4\" (UID: \"2999b63a-282f-4e12-9ca8-4004dc67a62f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3aa85e-vt4l4"
Oct 30 22:29:15.378320 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:15.378292    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7h86\" (UniqueName: \"kubernetes.io/projected/2999b63a-282f-4e12-9ca8-4004dc67a62f-kube-api-access-c7h86\") pod \"maas-api-revoke-keys-e2e-inf-a-3aa85e-vt4l4\" (UID: \"2999b63a-282f-4e12-9ca8-4004dc67a62f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3aa85e-vt4l4"
Oct 30 22:29:15.511329 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:15.511258    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3aa85e-vt4l4"
Oct 30 22:29:15.877190 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:15.876933    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3aa85e-vt4l4"]
Oct 30 22:29:15.896330 ip-10-0-130-60 kubenswrapper[5061]: W1030 22:29:15.896295    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2999b63a_282f_4e12_9ca8_4004dc67a62f.slice/crio-9af7364f384f77733e9a23344ccbeb4b1300f9f2fbf7309a031754b7e8a9b912 WatchSource:0}: Error finding container 9af7364f384f77733e9a23344ccbeb4b1300f9f2fbf7309a031754b7e8a9b912: Status 404 returned error can't find the container with id 9af7364f384f77733e9a23344ccbeb4b1300f9f2fbf7309a031754b7e8a9b912
Oct 30 22:29:16.126284 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:16.126251    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3aa85e-vt4l4" event={"ID":"2999b63a-282f-4e12-9ca8-4004dc67a62f","Type":"ContainerStarted","Data":"9af7364f384f77733e9a23344ccbeb4b1300f9f2fbf7309a031754b7e8a9b912"}
Oct 30 22:29:17.134467 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:17.134436    5061 generic.go:356] "Generic (PLEG): container finished" podID="2999b63a-282f-4e12-9ca8-4004dc67a62f" containerID="8020972ab398af65f33a70a827749f560ddbb4e2fada89888a5cd3543ecd0a65" exitCode=0
Oct 30 22:29:17.134866 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:17.134477    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3aa85e-vt4l4" event={"ID":"2999b63a-282f-4e12-9ca8-4004dc67a62f","Type":"ContainerDied","Data":"8020972ab398af65f33a70a827749f560ddbb4e2fada89888a5cd3543ecd0a65"}
Oct 30 22:29:18.284566 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:18.284514    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3aa85e-vt4l4"
Oct 30 22:29:18.357113 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:18.357084    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2999b63a-282f-4e12-9ca8-4004dc67a62f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2999b63a-282f-4e12-9ca8-4004dc67a62f-maas-api-openshift-ca\") pod \"2999b63a-282f-4e12-9ca8-4004dc67a62f\" (UID: \"2999b63a-282f-4e12-9ca8-4004dc67a62f\") "
Oct 30 22:29:18.357297 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:18.357121    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2999b63a-282f-4e12-9ca8-4004dc67a62f-kube-api-access-c7h86\" (UniqueName: \"kubernetes.io/projected/2999b63a-282f-4e12-9ca8-4004dc67a62f-kube-api-access-c7h86\") pod \"2999b63a-282f-4e12-9ca8-4004dc67a62f\" (UID: \"2999b63a-282f-4e12-9ca8-4004dc67a62f\") "
Oct 30 22:29:18.357297 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:18.357154    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2999b63a-282f-4e12-9ca8-4004dc67a62f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2999b63a-282f-4e12-9ca8-4004dc67a62f-maas-api-xks-ca\") pod \"2999b63a-282f-4e12-9ca8-4004dc67a62f\" (UID: \"2999b63a-282f-4e12-9ca8-4004dc67a62f\") "
Oct 30 22:29:18.357464 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:18.357441    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2999b63a-282f-4e12-9ca8-4004dc67a62f-maas-api-openshift-ca" pod "2999b63a-282f-4e12-9ca8-4004dc67a62f" (UID: "2999b63a-282f-4e12-9ca8-4004dc67a62f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:29:18.359268 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:18.359238    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2999b63a-282f-4e12-9ca8-4004dc67a62f-maas-api-xks-ca" pod "2999b63a-282f-4e12-9ca8-4004dc67a62f" (UID: "2999b63a-282f-4e12-9ca8-4004dc67a62f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:29:18.359388 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:18.359334    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2999b63a-282f-4e12-9ca8-4004dc67a62f-kube-api-access-c7h86" pod "2999b63a-282f-4e12-9ca8-4004dc67a62f" (UID: "2999b63a-282f-4e12-9ca8-4004dc67a62f"). InnerVolumeSpecName "kube-api-access-c7h86". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:29:18.458729 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:18.458628    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2999b63a-282f-4e12-9ca8-4004dc67a62f-maas-api-openshift-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:18.458729 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:18.458671    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7h86\" (UniqueName: \"kubernetes.io/projected/2999b63a-282f-4e12-9ca8-4004dc67a62f-kube-api-access-c7h86\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:18.458729 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:18.458682    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2999b63a-282f-4e12-9ca8-4004dc67a62f-maas-api-xks-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:19.147448 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:19.147414    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3aa85e-vt4l4"
Oct 30 22:29:19.147648 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:19.147450    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3aa85e-vt4l4" event={"ID":"2999b63a-282f-4e12-9ca8-4004dc67a62f","Type":"ContainerDied","Data":"9af7364f384f77733e9a23344ccbeb4b1300f9f2fbf7309a031754b7e8a9b912"}
Oct 30 22:29:19.147648 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:19.147500    5061 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9af7364f384f77733e9a23344ccbeb4b1300f9f2fbf7309a031754b7e8a9b912"
Oct 30 22:29:24.666860 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:29:24.666820    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 22:29:26.797972 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:26.797929    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-3aa85e-7865f7fb9f-cf7p2"]
Oct 30 22:29:27.051180 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:27.051095    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5632f8c-s797h"]
Oct 30 22:29:27.055447 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:27.055418    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4cbc3bf9-s8995"]
Oct 30 22:29:27.058448 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:27.058425    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5632f8c-s797h"]
Oct 30 22:29:27.084898 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:27.084869    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4cbc3bf9-s8995"]
Oct 30 22:29:28.528999 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:28.528965    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="932e2028-beda-405f-bc0a-6463a3cecac0" path="/var/lib/kubelet/pods/932e2028-beda-405f-bc0a-6463a3cecac0/volumes"
Oct 30 22:29:28.529478 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:28.529409    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc06e65c-d30e-4c21-9ee5-beb7e2e1c4de" path="/var/lib/kubelet/pods/fc06e65c-d30e-4c21-9ee5-beb7e2e1c4de/volumes"
Oct 30 22:29:29.201074 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:29.201042    5061 scope.go:122] "RemoveContainer" containerID="dce44c1fcb98d392020ac5b5f3e5412e255ed47bef466e5457f8cf4ae0f9f0bc"
Oct 30 22:29:29.210659 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:29.210640    5061 scope.go:122] "RemoveContainer" containerID="17dcecf330bfbdde1d1950d44502799ef7f5867ea2d505bf25aaeef3480433c3"
Oct 30 22:29:29.219682 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:29.219665    5061 scope.go:122] "RemoveContainer" containerID="ec31bf627ac29bb006d6ff6c1104005e6058a4567b86dfb7bf7eed885c05f73a"
Oct 30 22:29:29.227729 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:29.227712    5061 scope.go:122] "RemoveContainer" containerID="4afdfbc94d6c42f46da2ee0a8da31e7f99e48a3267eb9491dccbcb90a0ef160a"
Oct 30 22:29:29.235628 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:29.235598    5061 scope.go:122] "RemoveContainer" containerID="96619c69d6306a1554b4d5356b658d3d6ed92c4a7cd756fdbc41720fc21eea5f"
Oct 30 22:29:29.658671 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:29.658634    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-84a3cf/test-model-84a3cf-kserve-57554f46fc-j6rcb" podUID="8ca488a2-b652-49ff-a20a-17261a16a2bd" containerName="main" containerID="cri-o://7295e76cd512dac4ec5a11826d96c9fe5bfc2980dddea24f0bfc7252e18acec0" gracePeriod=45
Oct 30 22:29:29.906720 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:29.906693    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-84a3cf/test-model-84a3cf-kserve-57554f46fc-j6rcb"
Oct 30 22:29:29.978483 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:29.978391    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ca488a2-b652-49ff-a20a-17261a16a2bd-home\" (UniqueName: \"kubernetes.io/empty-dir/8ca488a2-b652-49ff-a20a-17261a16a2bd-home\") pod \"8ca488a2-b652-49ff-a20a-17261a16a2bd\" (UID: \"8ca488a2-b652-49ff-a20a-17261a16a2bd\") "
Oct 30 22:29:29.978483 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:29.978458    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ca488a2-b652-49ff-a20a-17261a16a2bd-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ca488a2-b652-49ff-a20a-17261a16a2bd-tmp-dir\") pod \"8ca488a2-b652-49ff-a20a-17261a16a2bd\" (UID: \"8ca488a2-b652-49ff-a20a-17261a16a2bd\") "
Oct 30 22:29:29.978758 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:29.978491    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ca488a2-b652-49ff-a20a-17261a16a2bd-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ca488a2-b652-49ff-a20a-17261a16a2bd-dshm\") pod \"8ca488a2-b652-49ff-a20a-17261a16a2bd\" (UID: \"8ca488a2-b652-49ff-a20a-17261a16a2bd\") "
Oct 30 22:29:29.978758 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:29.978588    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ca488a2-b652-49ff-a20a-17261a16a2bd-tls-certs\" (UniqueName: \"kubernetes.io/secret/8ca488a2-b652-49ff-a20a-17261a16a2bd-tls-certs\") pod \"8ca488a2-b652-49ff-a20a-17261a16a2bd\" (UID: \"8ca488a2-b652-49ff-a20a-17261a16a2bd\") "
Oct 30 22:29:29.978758 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:29.978648    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ca488a2-b652-49ff-a20a-17261a16a2bd-home" pod "8ca488a2-b652-49ff-a20a-17261a16a2bd" (UID: "8ca488a2-b652-49ff-a20a-17261a16a2bd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:29.978758 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:29.978654    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ca488a2-b652-49ff-a20a-17261a16a2bd-kube-api-access-xm5dg\" (UniqueName: \"kubernetes.io/projected/8ca488a2-b652-49ff-a20a-17261a16a2bd-kube-api-access-xm5dg\") pod \"8ca488a2-b652-49ff-a20a-17261a16a2bd\" (UID: \"8ca488a2-b652-49ff-a20a-17261a16a2bd\") "
Oct 30 22:29:29.978758 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:29.978740    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ca488a2-b652-49ff-a20a-17261a16a2bd-tmp-dir" pod "8ca488a2-b652-49ff-a20a-17261a16a2bd" (UID: "8ca488a2-b652-49ff-a20a-17261a16a2bd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:29.979039 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:29.978843    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ca488a2-b652-49ff-a20a-17261a16a2bd-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ca488a2-b652-49ff-a20a-17261a16a2bd-model-cache\") pod \"8ca488a2-b652-49ff-a20a-17261a16a2bd\" (UID: \"8ca488a2-b652-49ff-a20a-17261a16a2bd\") "
Oct 30 22:29:29.979039 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:29.979017    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ca488a2-b652-49ff-a20a-17261a16a2bd-model-cache" pod "8ca488a2-b652-49ff-a20a-17261a16a2bd" (UID: "8ca488a2-b652-49ff-a20a-17261a16a2bd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:29.979245 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:29.979221    5061 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ca488a2-b652-49ff-a20a-17261a16a2bd-tmp-dir\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:29.979245 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:29.979246    5061 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ca488a2-b652-49ff-a20a-17261a16a2bd-model-cache\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:29.979420 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:29.979259    5061 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ca488a2-b652-49ff-a20a-17261a16a2bd-home\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:29.980850 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:29.980822    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ca488a2-b652-49ff-a20a-17261a16a2bd-dshm" pod "8ca488a2-b652-49ff-a20a-17261a16a2bd" (UID: "8ca488a2-b652-49ff-a20a-17261a16a2bd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:29.980943 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:29.980861    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ca488a2-b652-49ff-a20a-17261a16a2bd-tls-certs" pod "8ca488a2-b652-49ff-a20a-17261a16a2bd" (UID: "8ca488a2-b652-49ff-a20a-17261a16a2bd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:29:29.980943 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:29.980897    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ca488a2-b652-49ff-a20a-17261a16a2bd-kube-api-access-xm5dg" pod "8ca488a2-b652-49ff-a20a-17261a16a2bd" (UID: "8ca488a2-b652-49ff-a20a-17261a16a2bd"). InnerVolumeSpecName "kube-api-access-xm5dg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:29:30.079883 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:30.079850    5061 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ca488a2-b652-49ff-a20a-17261a16a2bd-dshm\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:30.079883 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:30.079877    5061 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ca488a2-b652-49ff-a20a-17261a16a2bd-tls-certs\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:30.079883 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:30.079886    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xm5dg\" (UniqueName: \"kubernetes.io/projected/8ca488a2-b652-49ff-a20a-17261a16a2bd-kube-api-access-xm5dg\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:30.232429 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:30.232336    5061 generic.go:356] "Generic (PLEG): container finished" podID="8ca488a2-b652-49ff-a20a-17261a16a2bd" containerID="7295e76cd512dac4ec5a11826d96c9fe5bfc2980dddea24f0bfc7252e18acec0" exitCode=0
Oct 30 22:29:30.232429 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:30.232391    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-84a3cf/test-model-84a3cf-kserve-57554f46fc-j6rcb"
Oct 30 22:29:30.232680 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:30.232417    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-84a3cf/test-model-84a3cf-kserve-57554f46fc-j6rcb" event={"ID":"8ca488a2-b652-49ff-a20a-17261a16a2bd","Type":"ContainerDied","Data":"7295e76cd512dac4ec5a11826d96c9fe5bfc2980dddea24f0bfc7252e18acec0"}
Oct 30 22:29:30.232680 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:30.232467    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-84a3cf/test-model-84a3cf-kserve-57554f46fc-j6rcb" event={"ID":"8ca488a2-b652-49ff-a20a-17261a16a2bd","Type":"ContainerDied","Data":"2426080d2a521c92b37932e72b7988b4b001e954ababba8ab37d1bfbd0013b21"}
Oct 30 22:29:30.232680 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:30.232494    5061 scope.go:122] "RemoveContainer" containerID="7295e76cd512dac4ec5a11826d96c9fe5bfc2980dddea24f0bfc7252e18acec0"
Oct 30 22:29:30.242898 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:30.242881    5061 scope.go:122] "RemoveContainer" containerID="7295e76cd512dac4ec5a11826d96c9fe5bfc2980dddea24f0bfc7252e18acec0"
Oct 30 22:29:30.243152 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:29:30.243135    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7295e76cd512dac4ec5a11826d96c9fe5bfc2980dddea24f0bfc7252e18acec0\": container with ID starting with 7295e76cd512dac4ec5a11826d96c9fe5bfc2980dddea24f0bfc7252e18acec0 not found: ID does not exist" containerID="7295e76cd512dac4ec5a11826d96c9fe5bfc2980dddea24f0bfc7252e18acec0"
Oct 30 22:29:30.243196 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:30.243159    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7295e76cd512dac4ec5a11826d96c9fe5bfc2980dddea24f0bfc7252e18acec0"} err="failed to get container status \"7295e76cd512dac4ec5a11826d96c9fe5bfc2980dddea24f0bfc7252e18acec0\": rpc error: code = NotFound desc = could not find container \"7295e76cd512dac4ec5a11826d96c9fe5bfc2980dddea24f0bfc7252e18acec0\": container with ID starting with 7295e76cd512dac4ec5a11826d96c9fe5bfc2980dddea24f0bfc7252e18acec0 not found: ID does not exist"
Oct 30 22:29:30.387438 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:30.387405    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-84a3cf/test-model-84a3cf-kserve-57554f46fc-j6rcb"]
Oct 30 22:29:30.393797 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:30.393770    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-84a3cf/test-model-84a3cf-kserve-57554f46fc-j6rcb"]
Oct 30 22:29:30.528122 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:30.528050    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ca488a2-b652-49ff-a20a-17261a16a2bd" path="/var/lib/kubelet/pods/8ca488a2-b652-49ff-a20a-17261a16a2bd/volumes"
Oct 30 22:29:31.798251 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:31.798209    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3aa85e-7865f7fb9f-cf7p2" podUID="046f4c6f-bc56-4b85-af3e-f30e75964175" containerName="payload-pre-processing" containerID="cri-o://70ad6d5ec123f4f70fe6569ea774ff104f1e2f3edbae025eaabfe8d3855e3e02" gracePeriod=25
Oct 30 22:29:33.056378 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:33.056344    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47842a96-ccbdc"]
Oct 30 22:29:33.103399 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:33.103367    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47842a96-ccbdc"]
Oct 30 22:29:34.527357 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:34.527318    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3c3d477-f4c2-4740-b15e-aea2e058197b" path="/var/lib/kubelet/pods/e3c3d477-f4c2-4740-b15e-aea2e058197b/volumes"
Oct 30 22:29:37.291278 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:37.291240    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3aa85e-7865f7fb9f-cf7p2" podUID="046f4c6f-bc56-4b85-af3e-f30e75964175" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:29:46.516894 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:46.516807    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3aa85e-vt4l4"]
Oct 30 22:29:46.563628 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:46.563594    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3aa85e-vt4l4"]
Oct 30 22:29:46.573687 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:29:46.573643    5061 status_manager.go:1045] "Failed to get status for pod" err="pods \"maas-api-revoke-keys-e2e-inf-a-3aa85e-vt4l4\" is forbidden: User \"system:node:ip-10-0-130-60.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-130-60.us-west-2.compute.internal' and this object" podUID="2999b63a-282f-4e12-9ca8-4004dc67a62f" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3aa85e-vt4l4"
Oct 30 22:29:47.290672 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:47.290637    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3aa85e-7865f7fb9f-cf7p2" podUID="046f4c6f-bc56-4b85-af3e-f30e75964175" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:29:48.528194 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:48.528156    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2999b63a-282f-4e12-9ca8-4004dc67a62f" path="/var/lib/kubelet/pods/2999b63a-282f-4e12-9ca8-4004dc67a62f/volumes"
Oct 30 22:29:52.059165 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:52.059105    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3aa85e-7865f7fb9f-cf7p2"
Oct 30 22:29:52.113480 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:52.113450    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/046f4c6f-bc56-4b85-af3e-f30e75964175-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/046f4c6f-bc56-4b85-af3e-f30e75964175-plugins-config-volume\") pod \"046f4c6f-bc56-4b85-af3e-f30e75964175\" (UID: \"046f4c6f-bc56-4b85-af3e-f30e75964175\") "
Oct 30 22:29:52.113690 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:52.113497    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/046f4c6f-bc56-4b85-af3e-f30e75964175-kube-api-access-cb5pt\" (UniqueName: \"kubernetes.io/projected/046f4c6f-bc56-4b85-af3e-f30e75964175-kube-api-access-cb5pt\") pod \"046f4c6f-bc56-4b85-af3e-f30e75964175\" (UID: \"046f4c6f-bc56-4b85-af3e-f30e75964175\") "
Oct 30 22:29:52.113842 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:52.113820    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/046f4c6f-bc56-4b85-af3e-f30e75964175-plugins-config-volume" pod "046f4c6f-bc56-4b85-af3e-f30e75964175" (UID: "046f4c6f-bc56-4b85-af3e-f30e75964175"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:29:52.115861 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:52.115838    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/046f4c6f-bc56-4b85-af3e-f30e75964175-kube-api-access-cb5pt" pod "046f4c6f-bc56-4b85-af3e-f30e75964175" (UID: "046f4c6f-bc56-4b85-af3e-f30e75964175"). InnerVolumeSpecName "kube-api-access-cb5pt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:29:52.214626 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:52.214596    5061 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/046f4c6f-bc56-4b85-af3e-f30e75964175-plugins-config-volume\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:52.214626 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:52.214622    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cb5pt\" (UniqueName: \"kubernetes.io/projected/046f4c6f-bc56-4b85-af3e-f30e75964175-kube-api-access-cb5pt\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:52.366431 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:52.366342    5061 generic.go:356] "Generic (PLEG): container finished" podID="046f4c6f-bc56-4b85-af3e-f30e75964175" containerID="70ad6d5ec123f4f70fe6569ea774ff104f1e2f3edbae025eaabfe8d3855e3e02" exitCode=0
Oct 30 22:29:52.366431 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:52.366391    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3aa85e-7865f7fb9f-cf7p2"
Oct 30 22:29:52.366688 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:52.366429    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3aa85e-7865f7fb9f-cf7p2" event={"ID":"046f4c6f-bc56-4b85-af3e-f30e75964175","Type":"ContainerDied","Data":"70ad6d5ec123f4f70fe6569ea774ff104f1e2f3edbae025eaabfe8d3855e3e02"}
Oct 30 22:29:52.366688 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:52.366483    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3aa85e-7865f7fb9f-cf7p2" event={"ID":"046f4c6f-bc56-4b85-af3e-f30e75964175","Type":"ContainerDied","Data":"a51ee3229a475292cef484d040ae3c046ec4bbc57f2fe3e58cbbc3c97fc7d866"}
Oct 30 22:29:52.366688 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:52.366512    5061 scope.go:122] "RemoveContainer" containerID="70ad6d5ec123f4f70fe6569ea774ff104f1e2f3edbae025eaabfe8d3855e3e02"
Oct 30 22:29:52.376284 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:52.376265    5061 scope.go:122] "RemoveContainer" containerID="70ad6d5ec123f4f70fe6569ea774ff104f1e2f3edbae025eaabfe8d3855e3e02"
Oct 30 22:29:52.376577 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:29:52.376557    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70ad6d5ec123f4f70fe6569ea774ff104f1e2f3edbae025eaabfe8d3855e3e02\": container with ID starting with 70ad6d5ec123f4f70fe6569ea774ff104f1e2f3edbae025eaabfe8d3855e3e02 not found: ID does not exist" containerID="70ad6d5ec123f4f70fe6569ea774ff104f1e2f3edbae025eaabfe8d3855e3e02"
Oct 30 22:29:52.376630 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:52.376583    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70ad6d5ec123f4f70fe6569ea774ff104f1e2f3edbae025eaabfe8d3855e3e02"} err="failed to get container status \"70ad6d5ec123f4f70fe6569ea774ff104f1e2f3edbae025eaabfe8d3855e3e02\": rpc error: code = NotFound desc = could not find container \"70ad6d5ec123f4f70fe6569ea774ff104f1e2f3edbae025eaabfe8d3855e3e02\": container with ID starting with 70ad6d5ec123f4f70fe6569ea774ff104f1e2f3edbae025eaabfe8d3855e3e02 not found: ID does not exist"
Oct 30 22:29:52.520285 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:52.520251    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-3aa85e-7865f7fb9f-cf7p2"]
Oct 30 22:29:52.567238 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:52.567213    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-3aa85e-7865f7fb9f-cf7p2"]
Oct 30 22:29:54.041647 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:54.041615    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-84a3cf-ktgt6"]
Oct 30 22:29:54.042084 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:54.042071    5061 cpu_manager.go:414] "containerMap: removing container" podUID="8ca488a2-b652-49ff-a20a-17261a16a2bd" containerName="main"
Oct 30 22:29:54.042155 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:54.042085    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ca488a2-b652-49ff-a20a-17261a16a2bd" containerName="main"
Oct 30 22:29:54.042155 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:54.042115    5061 cpu_manager.go:414] "containerMap: removing container" podUID="046f4c6f-bc56-4b85-af3e-f30e75964175" containerName="payload-pre-processing"
Oct 30 22:29:54.042155 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:54.042120    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="046f4c6f-bc56-4b85-af3e-f30e75964175" containerName="payload-pre-processing"
Oct 30 22:29:54.042155 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:54.042136    5061 cpu_manager.go:414] "containerMap: removing container" podUID="2999b63a-282f-4e12-9ca8-4004dc67a62f" containerName="revoke-keys"
Oct 30 22:29:54.042155 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:54.042142    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2999b63a-282f-4e12-9ca8-4004dc67a62f" containerName="revoke-keys"
Oct 30 22:29:54.042316 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:54.042205    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="046f4c6f-bc56-4b85-af3e-f30e75964175" containerName="payload-pre-processing"
Oct 30 22:29:54.042316 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:54.042216    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="2999b63a-282f-4e12-9ca8-4004dc67a62f" containerName="revoke-keys"
Oct 30 22:29:54.042316 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:54.042224    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ca488a2-b652-49ff-a20a-17261a16a2bd" containerName="main"
Oct 30 22:29:54.047157 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:54.047137    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-84a3cf-ktgt6"
Oct 30 22:29:54.099163 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:54.099132    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:29:54.133596 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:54.133564    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8441ac0b-2d54-4374-8407-1d3ac661fc3e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-84a3cf-ktgt6\" (UID: \"8441ac0b-2d54-4374-8407-1d3ac661fc3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-84a3cf-ktgt6"
Oct 30 22:29:54.133728 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:54.133639    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwpng\" (UniqueName: \"kubernetes.io/projected/8441ac0b-2d54-4374-8407-1d3ac661fc3e-kube-api-access-rwpng\") pod \"maas-api-revoke-keys-e2e-inf-b-84a3cf-ktgt6\" (UID: \"8441ac0b-2d54-4374-8407-1d3ac661fc3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-84a3cf-ktgt6"
Oct 30 22:29:54.133770 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:54.133731    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8441ac0b-2d54-4374-8407-1d3ac661fc3e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-84a3cf-ktgt6\" (UID: \"8441ac0b-2d54-4374-8407-1d3ac661fc3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-84a3cf-ktgt6"
Oct 30 22:29:54.146245 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:54.146221    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:29:54.146376 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:54.146362    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:29:54.146462 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:54.146393    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gdmfg\""
Oct 30 22:29:54.148819 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:54.148799    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-84a3cf-ktgt6"]
Oct 30 22:29:54.235043 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:54.235014    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8441ac0b-2d54-4374-8407-1d3ac661fc3e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-84a3cf-ktgt6\" (UID: \"8441ac0b-2d54-4374-8407-1d3ac661fc3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-84a3cf-ktgt6"
Oct 30 22:29:54.235178 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:54.235090    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwpng\" (UniqueName: \"kubernetes.io/projected/8441ac0b-2d54-4374-8407-1d3ac661fc3e-kube-api-access-rwpng\") pod \"maas-api-revoke-keys-e2e-inf-b-84a3cf-ktgt6\" (UID: \"8441ac0b-2d54-4374-8407-1d3ac661fc3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-84a3cf-ktgt6"
Oct 30 22:29:54.235178 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:54.235117    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8441ac0b-2d54-4374-8407-1d3ac661fc3e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-84a3cf-ktgt6\" (UID: \"8441ac0b-2d54-4374-8407-1d3ac661fc3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-84a3cf-ktgt6"
Oct 30 22:29:54.235703 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:54.235684    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8441ac0b-2d54-4374-8407-1d3ac661fc3e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-84a3cf-ktgt6\" (UID: \"8441ac0b-2d54-4374-8407-1d3ac661fc3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-84a3cf-ktgt6"
Oct 30 22:29:54.237515 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:54.237492    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8441ac0b-2d54-4374-8407-1d3ac661fc3e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-84a3cf-ktgt6\" (UID: \"8441ac0b-2d54-4374-8407-1d3ac661fc3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-84a3cf-ktgt6"
Oct 30 22:29:54.289538 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:54.289506    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwpng\" (UniqueName: \"kubernetes.io/projected/8441ac0b-2d54-4374-8407-1d3ac661fc3e-kube-api-access-rwpng\") pod \"maas-api-revoke-keys-e2e-inf-b-84a3cf-ktgt6\" (UID: \"8441ac0b-2d54-4374-8407-1d3ac661fc3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-84a3cf-ktgt6"
Oct 30 22:29:54.359641 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:54.359610    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-84a3cf-ktgt6"
Oct 30 22:29:54.527571 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:54.527517    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="046f4c6f-bc56-4b85-af3e-f30e75964175" path="/var/lib/kubelet/pods/046f4c6f-bc56-4b85-af3e-f30e75964175/volumes"
Oct 30 22:29:54.666375 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:29:54.666287    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 22:29:54.729450 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:54.729418    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-84a3cf-ktgt6"]
Oct 30 22:29:54.730828 ip-10-0-130-60 kubenswrapper[5061]: W1030 22:29:54.730802    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8441ac0b_2d54_4374_8407_1d3ac661fc3e.slice/crio-c5edab4441e9645cf89eca138fdedf058ca45e71c33396b55c049d26bc321877 WatchSource:0}: Error finding container c5edab4441e9645cf89eca138fdedf058ca45e71c33396b55c049d26bc321877: Status 404 returned error can't find the container with id c5edab4441e9645cf89eca138fdedf058ca45e71c33396b55c049d26bc321877
Oct 30 22:29:55.392040 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:55.391997    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-84a3cf-ktgt6" event={"ID":"8441ac0b-2d54-4374-8407-1d3ac661fc3e","Type":"ContainerStarted","Data":"ab1ec9cad2b94c34faf4425e9b84ee08907baf0ccdb28aa5c0e5d207396fd2ca"}
Oct 30 22:29:55.392418 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:55.392065    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-84a3cf-ktgt6" event={"ID":"8441ac0b-2d54-4374-8407-1d3ac661fc3e","Type":"ContainerStarted","Data":"c5edab4441e9645cf89eca138fdedf058ca45e71c33396b55c049d26bc321877"}
Oct 30 22:29:55.502668 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:55.502616    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-84a3cf-ktgt6" podStartSLOduration=1.502602526 podStartE2EDuration="1.502602526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:29:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:29:55.500523515 +0000 UTC m=+2849.113293293" watchObservedRunningTime="2026-10-30 22:29:55.502602526 +0000 UTC m=+2849.115372283"
Oct 30 22:29:56.400126 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:56.400090    5061 generic.go:356] "Generic (PLEG): container finished" podID="8441ac0b-2d54-4374-8407-1d3ac661fc3e" containerID="ab1ec9cad2b94c34faf4425e9b84ee08907baf0ccdb28aa5c0e5d207396fd2ca" exitCode=0
Oct 30 22:29:56.400570 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:56.400169    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-84a3cf-ktgt6" event={"ID":"8441ac0b-2d54-4374-8407-1d3ac661fc3e","Type":"ContainerDied","Data":"ab1ec9cad2b94c34faf4425e9b84ee08907baf0ccdb28aa5c0e5d207396fd2ca"}
Oct 30 22:29:57.547977 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:57.547950    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-84a3cf-ktgt6"
Oct 30 22:29:57.669840 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:57.669756    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8441ac0b-2d54-4374-8407-1d3ac661fc3e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8441ac0b-2d54-4374-8407-1d3ac661fc3e-maas-api-openshift-ca\") pod \"8441ac0b-2d54-4374-8407-1d3ac661fc3e\" (UID: \"8441ac0b-2d54-4374-8407-1d3ac661fc3e\") "
Oct 30 22:29:57.670014 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:57.669849    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8441ac0b-2d54-4374-8407-1d3ac661fc3e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8441ac0b-2d54-4374-8407-1d3ac661fc3e-maas-api-xks-ca\") pod \"8441ac0b-2d54-4374-8407-1d3ac661fc3e\" (UID: \"8441ac0b-2d54-4374-8407-1d3ac661fc3e\") "
Oct 30 22:29:57.670014 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:57.669890    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8441ac0b-2d54-4374-8407-1d3ac661fc3e-kube-api-access-rwpng\" (UniqueName: \"kubernetes.io/projected/8441ac0b-2d54-4374-8407-1d3ac661fc3e-kube-api-access-rwpng\") pod \"8441ac0b-2d54-4374-8407-1d3ac661fc3e\" (UID: \"8441ac0b-2d54-4374-8407-1d3ac661fc3e\") "
Oct 30 22:29:57.670204 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:57.670178    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8441ac0b-2d54-4374-8407-1d3ac661fc3e-maas-api-openshift-ca" pod "8441ac0b-2d54-4374-8407-1d3ac661fc3e" (UID: "8441ac0b-2d54-4374-8407-1d3ac661fc3e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:29:57.672069 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:57.672035    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8441ac0b-2d54-4374-8407-1d3ac661fc3e-maas-api-xks-ca" pod "8441ac0b-2d54-4374-8407-1d3ac661fc3e" (UID: "8441ac0b-2d54-4374-8407-1d3ac661fc3e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:29:57.672177 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:57.672142    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8441ac0b-2d54-4374-8407-1d3ac661fc3e-kube-api-access-rwpng" pod "8441ac0b-2d54-4374-8407-1d3ac661fc3e" (UID: "8441ac0b-2d54-4374-8407-1d3ac661fc3e"). InnerVolumeSpecName "kube-api-access-rwpng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:29:57.770925 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:57.770892    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8441ac0b-2d54-4374-8407-1d3ac661fc3e-maas-api-openshift-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:57.770925 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:57.770919    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8441ac0b-2d54-4374-8407-1d3ac661fc3e-maas-api-xks-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:57.770925 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:57.770930    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwpng\" (UniqueName: \"kubernetes.io/projected/8441ac0b-2d54-4374-8407-1d3ac661fc3e-kube-api-access-rwpng\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:58.048124 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:58.048089    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75126b1f-hzlhv"]
Oct 30 22:29:58.050901 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:58.050875    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75126b1f-hzlhv"]
Oct 30 22:29:58.412866 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:58.412784    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-84a3cf-ktgt6"
Oct 30 22:29:58.412866 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:58.412811    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-84a3cf-ktgt6" event={"ID":"8441ac0b-2d54-4374-8407-1d3ac661fc3e","Type":"ContainerDied","Data":"c5edab4441e9645cf89eca138fdedf058ca45e71c33396b55c049d26bc321877"}
Oct 30 22:29:58.412866 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:58.412862    5061 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c5edab4441e9645cf89eca138fdedf058ca45e71c33396b55c049d26bc321877"
Oct 30 22:29:58.453468 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:29:58.453434    5061 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8441ac0b_2d54_4374_8407_1d3ac661fc3e.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 22:29:58.528045 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:58.528015    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17180b54-43c4-4af7-af97-ae8e37a2ba46" path="/var/lib/kubelet/pods/17180b54-43c4-4af7-af97-ae8e37a2ba46/volumes"
Oct 30 22:29:59.048122 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:59.048085    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8449ec64-vv9hj"]
Oct 30 22:29:59.095042 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:29:59.095007    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8449ec64-vv9hj"]
Oct 30 22:30:00.169353 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:00.169315    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-02bd3b-29889990-77nwk"]
Oct 30 22:30:00.170023 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:00.170000    5061 cpu_manager.go:414] "containerMap: removing container" podUID="8441ac0b-2d54-4374-8407-1d3ac661fc3e" containerName="revoke-keys"
Oct 30 22:30:00.170119 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:00.170025    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8441ac0b-2d54-4374-8407-1d3ac661fc3e" containerName="revoke-keys"
Oct 30 22:30:00.170202 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:00.170189    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="8441ac0b-2d54-4374-8407-1d3ac661fc3e" containerName="revoke-keys"
Oct 30 22:30:00.173795 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:00.173771    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5402dd-29889990-45272"]
Oct 30 22:30:00.173926 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:00.173885    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-02bd3b-29889990-77nwk"
Oct 30 22:30:00.177450 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:00.177427    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-4d8041-29889990-6zvnx"]
Oct 30 22:30:00.177572 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:00.177520    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5402dd-29889990-45272"
Oct 30 22:30:00.180916 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:00.180897    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-4d8041-29889990-6zvnx"
Oct 30 22:30:00.217594 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:00.217570    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-84a3cf-29889990-njn48"]
Oct 30 22:30:00.221826 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:00.221808    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-84a3cf-29889990-njn48"
Oct 30 22:30:00.223927 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:00.223904    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:30:00.224026 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:00.223929    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:30:00.224026 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:00.223910    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gdmfg\""
Oct 30 22:30:00.267554 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:00.267508    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5402dd-29889990-45272"]
Oct 30 22:30:00.272260 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:00.272234    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-02bd3b-29889990-77nwk"]
Oct 30 22:30:00.298271 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:00.298238    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51015ddd-3122-4cfb-99bc-d4c041646c64-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-4d8041-29889990-6zvnx\" (UID: \"51015ddd-3122-4cfb-99bc-d4c041646c64\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-4d8041-29889990-6zvnx"
Oct 30 22:30:00.298271 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:00.298272    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/333a34f7-9655-4ea6-9588-b9f5cb8fe305-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-02bd3b-29889990-77nwk\" (UID: \"333a34f7-9655-4ea6-9588-b9f5cb8fe305\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-02bd3b-29889990-77nwk"
Oct 30 22:30:00.298516 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:00.298298    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jbkr\" (UniqueName: \"kubernetes.io/projected/d14224db-1f4f-45c3-a36b-8af75018a006-kube-api-access-6jbkr\") pod \"maas-api-key-cleanup-e2e-worker-w5-5402dd-29889990-45272\" (UID: \"d14224db-1f4f-45c3-a36b-8af75018a006\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5402dd-29889990-45272"
Oct 30 22:30:00.298516 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:00.298416    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2jl4\" (UniqueName: \"kubernetes.io/projected/51015ddd-3122-4cfb-99bc-d4c041646c64-kube-api-access-s2jl4\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-4d8041-29889990-6zvnx\" (UID: \"51015ddd-3122-4cfb-99bc-d4c041646c64\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-4d8041-29889990-6zvnx"
Oct 30 22:30:00.298516 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:00.298458    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cf662ac-1070-4ee6-bf5d-f117beda9c6d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-84a3cf-29889990-njn48\" (UID: \"7cf662ac-1070-4ee6-bf5d-f117beda9c6d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-84a3cf-29889990-njn48"
Oct 30 22:30:00.298516 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:00.298507    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d14224db-1f4f-45c3-a36b-8af75018a006-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-5402dd-29889990-45272\" (UID: \"d14224db-1f4f-45c3-a36b-8af75018a006\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5402dd-29889990-45272"
Oct 30 22:30:00.298787 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:00.298563    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrrql\" (UniqueName: \"kubernetes.io/projected/333a34f7-9655-4ea6-9588-b9f5cb8fe305-kube-api-access-vrrql\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-02bd3b-29889990-77nwk\" (UID: \"333a34f7-9655-4ea6-9588-b9f5cb8fe305\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-02bd3b-29889990-77nwk"
Oct 30 22:30:00.298787 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:00.298650    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45rsx\" (UniqueName: \"kubernetes.io/projected/7cf662ac-1070-4ee6-bf5d-f117beda9c6d-kube-api-access-45rsx\") pod \"maas-api-key-cleanup-e2e-inf-b-84a3cf-29889990-njn48\" (UID: \"7cf662ac-1070-4ee6-bf5d-f117beda9c6d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-84a3cf-29889990-njn48"
Oct 30 22:30:00.307552 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:00.307507    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-84a3cf-29889990-njn48"]
Oct 30 22:30:00.309834 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:00.309815    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-4d8041-29889990-6zvnx"]
Oct 30 22:30:00.400084 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:00.400046    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51015ddd-3122-4cfb-99bc-d4c041646c64-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-4d8041-29889990-6zvnx\" (UID: \"51015ddd-3122-4cfb-99bc-d4c041646c64\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-4d8041-29889990-6zvnx"
Oct 30 22:30:00.400084 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:00.400085    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/333a34f7-9655-4ea6-9588-b9f5cb8fe305-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-02bd3b-29889990-77nwk\" (UID: \"333a34f7-9655-4ea6-9588-b9f5cb8fe305\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-02bd3b-29889990-77nwk"
Oct 30 22:30:00.400344 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:00.400117    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jbkr\" (UniqueName: \"kubernetes.io/projected/d14224db-1f4f-45c3-a36b-8af75018a006-kube-api-access-6jbkr\") pod \"maas-api-key-cleanup-e2e-worker-w5-5402dd-29889990-45272\" (UID: \"d14224db-1f4f-45c3-a36b-8af75018a006\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5402dd-29889990-45272"
Oct 30 22:30:00.400344 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:00.400146    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2jl4\" (UniqueName: \"kubernetes.io/projected/51015ddd-3122-4cfb-99bc-d4c041646c64-kube-api-access-s2jl4\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-4d8041-29889990-6zvnx\" (UID: \"51015ddd-3122-4cfb-99bc-d4c041646c64\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-4d8041-29889990-6zvnx"
Oct 30 22:30:00.400344 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:00.400174    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cf662ac-1070-4ee6-bf5d-f117beda9c6d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-84a3cf-29889990-njn48\" (UID: \"7cf662ac-1070-4ee6-bf5d-f117beda9c6d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-84a3cf-29889990-njn48"
Oct 30 22:30:00.400344 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:00.400200    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d14224db-1f4f-45c3-a36b-8af75018a006-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-5402dd-29889990-45272\" (UID: \"d14224db-1f4f-45c3-a36b-8af75018a006\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5402dd-29889990-45272"
Oct 30 22:30:00.400344 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:00.400244    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrrql\" (UniqueName: \"kubernetes.io/projected/333a34f7-9655-4ea6-9588-b9f5cb8fe305-kube-api-access-vrrql\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-02bd3b-29889990-77nwk\" (UID: \"333a34f7-9655-4ea6-9588-b9f5cb8fe305\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-02bd3b-29889990-77nwk"
Oct 30 22:30:00.400344 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:00.400320    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45rsx\" (UniqueName: \"kubernetes.io/projected/7cf662ac-1070-4ee6-bf5d-f117beda9c6d-kube-api-access-45rsx\") pod \"maas-api-key-cleanup-e2e-inf-b-84a3cf-29889990-njn48\" (UID: \"7cf662ac-1070-4ee6-bf5d-f117beda9c6d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-84a3cf-29889990-njn48"
Oct 30 22:30:00.400885 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:00.400861    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/333a34f7-9655-4ea6-9588-b9f5cb8fe305-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-02bd3b-29889990-77nwk\" (UID: \"333a34f7-9655-4ea6-9588-b9f5cb8fe305\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-02bd3b-29889990-77nwk"
Oct 30 22:30:00.400983 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:00.400963    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d14224db-1f4f-45c3-a36b-8af75018a006-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-5402dd-29889990-45272\" (UID: \"d14224db-1f4f-45c3-a36b-8af75018a006\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5402dd-29889990-45272"
Oct 30 22:30:00.401051 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:00.401009    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51015ddd-3122-4cfb-99bc-d4c041646c64-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-4d8041-29889990-6zvnx\" (UID: \"51015ddd-3122-4cfb-99bc-d4c041646c64\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-4d8041-29889990-6zvnx"
Oct 30 22:30:00.401051 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:00.401007    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cf662ac-1070-4ee6-bf5d-f117beda9c6d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-84a3cf-29889990-njn48\" (UID: \"7cf662ac-1070-4ee6-bf5d-f117beda9c6d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-84a3cf-29889990-njn48"
Oct 30 22:30:00.455102 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:00.455028    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jbkr\" (UniqueName: \"kubernetes.io/projected/d14224db-1f4f-45c3-a36b-8af75018a006-kube-api-access-6jbkr\") pod \"maas-api-key-cleanup-e2e-worker-w5-5402dd-29889990-45272\" (UID: \"d14224db-1f4f-45c3-a36b-8af75018a006\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5402dd-29889990-45272"
Oct 30 22:30:00.455236 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:00.455154    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45rsx\" (UniqueName: \"kubernetes.io/projected/7cf662ac-1070-4ee6-bf5d-f117beda9c6d-kube-api-access-45rsx\") pod \"maas-api-key-cleanup-e2e-inf-b-84a3cf-29889990-njn48\" (UID: \"7cf662ac-1070-4ee6-bf5d-f117beda9c6d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-84a3cf-29889990-njn48"
Oct 30 22:30:00.455236 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:00.455158    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrrql\" (UniqueName: \"kubernetes.io/projected/333a34f7-9655-4ea6-9588-b9f5cb8fe305-kube-api-access-vrrql\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-02bd3b-29889990-77nwk\" (UID: \"333a34f7-9655-4ea6-9588-b9f5cb8fe305\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-02bd3b-29889990-77nwk"
Oct 30 22:30:00.455236 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:00.455200    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2jl4\" (UniqueName: \"kubernetes.io/projected/51015ddd-3122-4cfb-99bc-d4c041646c64-kube-api-access-s2jl4\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-4d8041-29889990-6zvnx\" (UID: \"51015ddd-3122-4cfb-99bc-d4c041646c64\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-4d8041-29889990-6zvnx"
Oct 30 22:30:00.490473 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:00.490442    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-02bd3b-29889990-77nwk"
Oct 30 22:30:00.498424 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:00.498399    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5402dd-29889990-45272"
Oct 30 22:30:00.504643 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:00.504470    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-4d8041-29889990-6zvnx"
Oct 30 22:30:00.533862 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:00.533825    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86f24e81-ee68-4d6f-b1f7-c55ff25c872d" path="/var/lib/kubelet/pods/86f24e81-ee68-4d6f-b1f7-c55ff25c872d/volumes"
Oct 30 22:30:00.534433 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:00.534410    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-84a3cf-29889990-njn48"
Oct 30 22:30:00.870613 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:00.870579    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-02bd3b-29889990-77nwk"]
Oct 30 22:30:00.871943 ip-10-0-130-60 kubenswrapper[5061]: W1030 22:30:00.871914    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod333a34f7_9655_4ea6_9588_b9f5cb8fe305.slice/crio-65571dbe669318b99a6c4f6998f53baf78600abb5b38bddbfacb147f8912f6ad WatchSource:0}: Error finding container 65571dbe669318b99a6c4f6998f53baf78600abb5b38bddbfacb147f8912f6ad: Status 404 returned error can't find the container with id 65571dbe669318b99a6c4f6998f53baf78600abb5b38bddbfacb147f8912f6ad
Oct 30 22:30:00.917837 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:00.917814    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-4d8041-29889990-6zvnx"]
Oct 30 22:30:00.918675 ip-10-0-130-60 kubenswrapper[5061]: W1030 22:30:00.918651    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51015ddd_3122_4cfb_99bc_d4c041646c64.slice/crio-7b6fa411593fb1f3d4c4b20ff12e5a89f797bc88f1dd9780e7e23f65ce12e89d WatchSource:0}: Error finding container 7b6fa411593fb1f3d4c4b20ff12e5a89f797bc88f1dd9780e7e23f65ce12e89d: Status 404 returned error can't find the container with id 7b6fa411593fb1f3d4c4b20ff12e5a89f797bc88f1dd9780e7e23f65ce12e89d
Oct 30 22:30:00.920810 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:00.920789    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5402dd-29889990-45272"]
Oct 30 22:30:00.921883 ip-10-0-130-60 kubenswrapper[5061]: W1030 22:30:00.921797    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd14224db_1f4f_45c3_a36b_8af75018a006.slice/crio-0a0e3101419fb3302cb98f90bada528ef3bedc5e18cf1c63cd20b698ca63aae9 WatchSource:0}: Error finding container 0a0e3101419fb3302cb98f90bada528ef3bedc5e18cf1c63cd20b698ca63aae9: Status 404 returned error can't find the container with id 0a0e3101419fb3302cb98f90bada528ef3bedc5e18cf1c63cd20b698ca63aae9
Oct 30 22:30:00.965195 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:00.965172    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-84a3cf-29889990-njn48"]
Oct 30 22:30:00.966960 ip-10-0-130-60 kubenswrapper[5061]: W1030 22:30:00.966933    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cf662ac_1070_4ee6_bf5d_f117beda9c6d.slice/crio-fdf3ff41a57be4339dc2d5dac351e18b733cb534b75cfedd7035e4c15fe7aa82 WatchSource:0}: Error finding container fdf3ff41a57be4339dc2d5dac351e18b733cb534b75cfedd7035e4c15fe7aa82: Status 404 returned error can't find the container with id fdf3ff41a57be4339dc2d5dac351e18b733cb534b75cfedd7035e4c15fe7aa82
Oct 30 22:30:01.431385 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:01.431291    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-4d8041-29889990-6zvnx" event={"ID":"51015ddd-3122-4cfb-99bc-d4c041646c64","Type":"ContainerStarted","Data":"153fef98655a7b76b948be7c0bfaf653a06532d9c7378a0e2f3b63a34b7e8c0f"}
Oct 30 22:30:01.431385 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:01.431358    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-4d8041-29889990-6zvnx" event={"ID":"51015ddd-3122-4cfb-99bc-d4c041646c64","Type":"ContainerStarted","Data":"7b6fa411593fb1f3d4c4b20ff12e5a89f797bc88f1dd9780e7e23f65ce12e89d"}
Oct 30 22:30:01.432875 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:01.432849    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-02bd3b-29889990-77nwk" event={"ID":"333a34f7-9655-4ea6-9588-b9f5cb8fe305","Type":"ContainerStarted","Data":"19056e05032c34ab1bcb65918632651de51adb8047bec036b21bc21754105528"}
Oct 30 22:30:01.432988 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:01.432896    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-02bd3b-29889990-77nwk" event={"ID":"333a34f7-9655-4ea6-9588-b9f5cb8fe305","Type":"ContainerStarted","Data":"65571dbe669318b99a6c4f6998f53baf78600abb5b38bddbfacb147f8912f6ad"}
Oct 30 22:30:01.434360 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:01.434331    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5402dd-29889990-45272" event={"ID":"d14224db-1f4f-45c3-a36b-8af75018a006","Type":"ContainerStarted","Data":"10a4471ea7f5b56906a3d805047b9a4495a695729f1b700dbcc65e50b06bce9d"}
Oct 30 22:30:01.434469 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:01.434370    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5402dd-29889990-45272" event={"ID":"d14224db-1f4f-45c3-a36b-8af75018a006","Type":"ContainerStarted","Data":"0a0e3101419fb3302cb98f90bada528ef3bedc5e18cf1c63cd20b698ca63aae9"}
Oct 30 22:30:01.435716 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:01.435695    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-84a3cf-29889990-njn48" event={"ID":"7cf662ac-1070-4ee6-bf5d-f117beda9c6d","Type":"ContainerStarted","Data":"cccacdbdab4629ec3c163c02e5c74e98765904677fcab5da90158cf473861501"}
Oct 30 22:30:01.435812 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:01.435733    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-84a3cf-29889990-njn48" event={"ID":"7cf662ac-1070-4ee6-bf5d-f117beda9c6d","Type":"ContainerStarted","Data":"fdf3ff41a57be4339dc2d5dac351e18b733cb534b75cfedd7035e4c15fe7aa82"}
Oct 30 22:30:01.646966 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:01.646905    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-4d8041-29889990-6zvnx" podStartSLOduration=1.6468894509999998 podStartE2EDuration="1.646889451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:01.538204543 +0000 UTC m=+2855.150974320" watchObservedRunningTime="2026-10-30 22:30:01.646889451 +0000 UTC m=+2855.259659208"
Oct 30 22:30:01.648282 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:01.648246    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-02bd3b-29889990-77nwk" podStartSLOduration=1.648234419 podStartE2EDuration="1.648234419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:01.645464403 +0000 UTC m=+2855.258234161" watchObservedRunningTime="2026-10-30 22:30:01.648234419 +0000 UTC m=+2855.261004173"
Oct 30 22:30:01.754094 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:01.754042    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5402dd-29889990-45272" podStartSLOduration=1.7540273100000001 podStartE2EDuration="1.75402731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:01.753512572 +0000 UTC m=+2855.366282331" watchObservedRunningTime="2026-10-30 22:30:01.75402731 +0000 UTC m=+2855.366797088"
Oct 30 22:30:01.864825 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:01.864768    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-84a3cf-29889990-njn48" podStartSLOduration=1.864753272 podStartE2EDuration="1.864753272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:01.862841249 +0000 UTC m=+2855.475611026" watchObservedRunningTime="2026-10-30 22:30:01.864753272 +0000 UTC m=+2855.477523029"
Oct 30 22:30:02.443755 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:02.443722    5061 generic.go:356] "Generic (PLEG): container finished" podID="51015ddd-3122-4cfb-99bc-d4c041646c64" containerID="153fef98655a7b76b948be7c0bfaf653a06532d9c7378a0e2f3b63a34b7e8c0f" exitCode=60
Oct 30 22:30:02.444196 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:02.443808    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-4d8041-29889990-6zvnx" event={"ID":"51015ddd-3122-4cfb-99bc-d4c041646c64","Type":"ContainerDied","Data":"153fef98655a7b76b948be7c0bfaf653a06532d9c7378a0e2f3b63a34b7e8c0f"}
Oct 30 22:30:02.444196 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:02.444074    5061 scope.go:122] "RemoveContainer" containerID="153fef98655a7b76b948be7c0bfaf653a06532d9c7378a0e2f3b63a34b7e8c0f"
Oct 30 22:30:02.445569 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:02.445547    5061 generic.go:356] "Generic (PLEG): container finished" podID="333a34f7-9655-4ea6-9588-b9f5cb8fe305" containerID="19056e05032c34ab1bcb65918632651de51adb8047bec036b21bc21754105528" exitCode=60
Oct 30 22:30:02.445652 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:02.445630    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-02bd3b-29889990-77nwk" event={"ID":"333a34f7-9655-4ea6-9588-b9f5cb8fe305","Type":"ContainerDied","Data":"19056e05032c34ab1bcb65918632651de51adb8047bec036b21bc21754105528"}
Oct 30 22:30:02.445835 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:02.445817    5061 scope.go:122] "RemoveContainer" containerID="19056e05032c34ab1bcb65918632651de51adb8047bec036b21bc21754105528"
Oct 30 22:30:02.447307 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:02.447286    5061 generic.go:356] "Generic (PLEG): container finished" podID="d14224db-1f4f-45c3-a36b-8af75018a006" containerID="10a4471ea7f5b56906a3d805047b9a4495a695729f1b700dbcc65e50b06bce9d" exitCode=60
Oct 30 22:30:02.447401 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:02.447366    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5402dd-29889990-45272" event={"ID":"d14224db-1f4f-45c3-a36b-8af75018a006","Type":"ContainerDied","Data":"10a4471ea7f5b56906a3d805047b9a4495a695729f1b700dbcc65e50b06bce9d"}
Oct 30 22:30:02.447564 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:02.447546    5061 scope.go:122] "RemoveContainer" containerID="10a4471ea7f5b56906a3d805047b9a4495a695729f1b700dbcc65e50b06bce9d"
Oct 30 22:30:02.448940 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:02.448921    5061 generic.go:356] "Generic (PLEG): container finished" podID="7cf662ac-1070-4ee6-bf5d-f117beda9c6d" containerID="cccacdbdab4629ec3c163c02e5c74e98765904677fcab5da90158cf473861501" exitCode=60
Oct 30 22:30:02.449020 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:02.448946    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-84a3cf-29889990-njn48" event={"ID":"7cf662ac-1070-4ee6-bf5d-f117beda9c6d","Type":"ContainerDied","Data":"cccacdbdab4629ec3c163c02e5c74e98765904677fcab5da90158cf473861501"}
Oct 30 22:30:02.449175 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:02.449159    5061 scope.go:122] "RemoveContainer" containerID="cccacdbdab4629ec3c163c02e5c74e98765904677fcab5da90158cf473861501"
Oct 30 22:30:03.457284 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:03.457248    5061 generic.go:356] "Generic (PLEG): container finished" podID="333a34f7-9655-4ea6-9588-b9f5cb8fe305" containerID="32bbac1f6f4207ab4251606419130d5dfa5fa39f6ca2bed518dcab46ddf66f7c" exitCode=60
Oct 30 22:30:03.457715 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:03.457330    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-02bd3b-29889990-77nwk" event={"ID":"333a34f7-9655-4ea6-9588-b9f5cb8fe305","Type":"ContainerDied","Data":"32bbac1f6f4207ab4251606419130d5dfa5fa39f6ca2bed518dcab46ddf66f7c"}
Oct 30 22:30:03.457715 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:03.457389    5061 scope.go:122] "RemoveContainer" containerID="19056e05032c34ab1bcb65918632651de51adb8047bec036b21bc21754105528"
Oct 30 22:30:03.457715 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:03.457679    5061 scope.go:122] "RemoveContainer" containerID="32bbac1f6f4207ab4251606419130d5dfa5fa39f6ca2bed518dcab46ddf66f7c"
Oct 30 22:30:03.458009 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:30:03.457986    5061 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-02bd3b-29889990-77nwk_odh-ai-gateway-infra(333a34f7-9655-4ea6-9588-b9f5cb8fe305)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-02bd3b-29889990-77nwk" podUID="333a34f7-9655-4ea6-9588-b9f5cb8fe305"
Oct 30 22:30:03.459615 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:03.459585    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5402dd-29889990-45272" event={"ID":"d14224db-1f4f-45c3-a36b-8af75018a006","Type":"ContainerStarted","Data":"8872f3be6cc4c7a793c1a0c99d3acb82488ece9ae36963bbb2f30a3550aa0f46"}
Oct 30 22:30:03.461381 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:03.461355    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-84a3cf-29889990-njn48" event={"ID":"7cf662ac-1070-4ee6-bf5d-f117beda9c6d","Type":"ContainerStarted","Data":"a47f1cbfe43da2bba66e716f3ab892bb0a148c73715c5b68b0f96ede014e8c39"}
Oct 30 22:30:03.463255 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:03.463233    5061 generic.go:356] "Generic (PLEG): container finished" podID="51015ddd-3122-4cfb-99bc-d4c041646c64" containerID="100e3948fac4a85a1a6d838aea832ef4564d6d90aa52c0ac74fa79a98339714a" exitCode=60
Oct 30 22:30:03.463354 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:03.463274    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-4d8041-29889990-6zvnx" event={"ID":"51015ddd-3122-4cfb-99bc-d4c041646c64","Type":"ContainerDied","Data":"100e3948fac4a85a1a6d838aea832ef4564d6d90aa52c0ac74fa79a98339714a"}
Oct 30 22:30:03.463485 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:03.463472    5061 scope.go:122] "RemoveContainer" containerID="100e3948fac4a85a1a6d838aea832ef4564d6d90aa52c0ac74fa79a98339714a"
Oct 30 22:30:03.463693 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:30:03.463671    5061 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-4d8041-29889990-6zvnx_odh-ai-gateway-infra(51015ddd-3122-4cfb-99bc-d4c041646c64)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-4d8041-29889990-6zvnx" podUID="51015ddd-3122-4cfb-99bc-d4c041646c64"
Oct 30 22:30:03.632355 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:03.632326    5061 scope.go:122] "RemoveContainer" containerID="153fef98655a7b76b948be7c0bfaf653a06532d9c7378a0e2f3b63a34b7e8c0f"
Oct 30 22:30:04.472101 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:04.472071    5061 scope.go:122] "RemoveContainer" containerID="32bbac1f6f4207ab4251606419130d5dfa5fa39f6ca2bed518dcab46ddf66f7c"
Oct 30 22:30:04.472595 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:30:04.472329    5061 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-02bd3b-29889990-77nwk_odh-ai-gateway-infra(333a34f7-9655-4ea6-9588-b9f5cb8fe305)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-02bd3b-29889990-77nwk" podUID="333a34f7-9655-4ea6-9588-b9f5cb8fe305"
Oct 30 22:30:04.473885 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:04.473860    5061 generic.go:356] "Generic (PLEG): container finished" podID="d14224db-1f4f-45c3-a36b-8af75018a006" containerID="8872f3be6cc4c7a793c1a0c99d3acb82488ece9ae36963bbb2f30a3550aa0f46" exitCode=60
Oct 30 22:30:04.473994 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:04.473950    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5402dd-29889990-45272" event={"ID":"d14224db-1f4f-45c3-a36b-8af75018a006","Type":"ContainerDied","Data":"8872f3be6cc4c7a793c1a0c99d3acb82488ece9ae36963bbb2f30a3550aa0f46"}
Oct 30 22:30:04.474075 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:04.474013    5061 scope.go:122] "RemoveContainer" containerID="10a4471ea7f5b56906a3d805047b9a4495a695729f1b700dbcc65e50b06bce9d"
Oct 30 22:30:04.474147 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:04.474129    5061 scope.go:122] "RemoveContainer" containerID="8872f3be6cc4c7a793c1a0c99d3acb82488ece9ae36963bbb2f30a3550aa0f46"
Oct 30 22:30:04.474410 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:30:04.474391    5061 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-5402dd-29889990-45272_odh-ai-gateway-infra(d14224db-1f4f-45c3-a36b-8af75018a006)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5402dd-29889990-45272" podUID="d14224db-1f4f-45c3-a36b-8af75018a006"
Oct 30 22:30:04.475768 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:04.475745    5061 generic.go:356] "Generic (PLEG): container finished" podID="7cf662ac-1070-4ee6-bf5d-f117beda9c6d" containerID="a47f1cbfe43da2bba66e716f3ab892bb0a148c73715c5b68b0f96ede014e8c39" exitCode=60
Oct 30 22:30:04.475881 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:04.475827    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-84a3cf-29889990-njn48" event={"ID":"7cf662ac-1070-4ee6-bf5d-f117beda9c6d","Type":"ContainerDied","Data":"a47f1cbfe43da2bba66e716f3ab892bb0a148c73715c5b68b0f96ede014e8c39"}
Oct 30 22:30:04.476004 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:04.475985    5061 scope.go:122] "RemoveContainer" containerID="a47f1cbfe43da2bba66e716f3ab892bb0a148c73715c5b68b0f96ede014e8c39"
Oct 30 22:30:04.476244 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:30:04.476223    5061 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-84a3cf-29889990-njn48_odh-ai-gateway-infra(7cf662ac-1070-4ee6-bf5d-f117beda9c6d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-84a3cf-29889990-njn48" podUID="7cf662ac-1070-4ee6-bf5d-f117beda9c6d"
Oct 30 22:30:04.477611 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:04.477593    5061 scope.go:122] "RemoveContainer" containerID="100e3948fac4a85a1a6d838aea832ef4564d6d90aa52c0ac74fa79a98339714a"
Oct 30 22:30:04.477788 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:30:04.477770    5061 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-4d8041-29889990-6zvnx_odh-ai-gateway-infra(51015ddd-3122-4cfb-99bc-d4c041646c64)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-4d8041-29889990-6zvnx" podUID="51015ddd-3122-4cfb-99bc-d4c041646c64"
Oct 30 22:30:04.487097 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:04.487072    5061 scope.go:122] "RemoveContainer" containerID="cccacdbdab4629ec3c163c02e5c74e98765904677fcab5da90158cf473861501"
Oct 30 22:30:05.486578 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:05.486551    5061 scope.go:122] "RemoveContainer" containerID="8872f3be6cc4c7a793c1a0c99d3acb82488ece9ae36963bbb2f30a3550aa0f46"
Oct 30 22:30:05.487018 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:30:05.486850    5061 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-5402dd-29889990-45272_odh-ai-gateway-infra(d14224db-1f4f-45c3-a36b-8af75018a006)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5402dd-29889990-45272" podUID="d14224db-1f4f-45c3-a36b-8af75018a006"
Oct 30 22:30:05.488041 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:05.488021    5061 scope.go:122] "RemoveContainer" containerID="a47f1cbfe43da2bba66e716f3ab892bb0a148c73715c5b68b0f96ede014e8c39"
Oct 30 22:30:05.488225 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:30:05.488209    5061 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-84a3cf-29889990-njn48_odh-ai-gateway-infra(7cf662ac-1070-4ee6-bf5d-f117beda9c6d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-84a3cf-29889990-njn48" podUID="7cf662ac-1070-4ee6-bf5d-f117beda9c6d"
Oct 30 22:30:05.585990 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:05.585952    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-84a3cf-6c95488bd4-cjmzh"]
Oct 30 22:30:05.690426 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:05.690391    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-84a3cf-6cb65fb6cf-r64nq"]
Oct 30 22:30:10.208519 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:10.208483    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-84a3cf-29889990-njn48"]
Oct 30 22:30:10.365518 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:10.365494    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-84a3cf-29889990-njn48"
Oct 30 22:30:10.503416 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:10.503339    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7cf662ac-1070-4ee6-bf5d-f117beda9c6d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cf662ac-1070-4ee6-bf5d-f117beda9c6d-maas-api-service-ca\") pod \"7cf662ac-1070-4ee6-bf5d-f117beda9c6d\" (UID: \"7cf662ac-1070-4ee6-bf5d-f117beda9c6d\") "
Oct 30 22:30:10.503626 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:10.503418    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cf662ac-1070-4ee6-bf5d-f117beda9c6d-kube-api-access-45rsx\" (UniqueName: \"kubernetes.io/projected/7cf662ac-1070-4ee6-bf5d-f117beda9c6d-kube-api-access-45rsx\") pod \"7cf662ac-1070-4ee6-bf5d-f117beda9c6d\" (UID: \"7cf662ac-1070-4ee6-bf5d-f117beda9c6d\") "
Oct 30 22:30:10.503768 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:10.503744    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cf662ac-1070-4ee6-bf5d-f117beda9c6d-maas-api-service-ca" pod "7cf662ac-1070-4ee6-bf5d-f117beda9c6d" (UID: "7cf662ac-1070-4ee6-bf5d-f117beda9c6d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:10.505493 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:10.505470    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cf662ac-1070-4ee6-bf5d-f117beda9c6d-kube-api-access-45rsx" pod "7cf662ac-1070-4ee6-bf5d-f117beda9c6d" (UID: "7cf662ac-1070-4ee6-bf5d-f117beda9c6d"). InnerVolumeSpecName "kube-api-access-45rsx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:10.520300 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:10.520274    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-84a3cf-29889990-njn48" event={"ID":"7cf662ac-1070-4ee6-bf5d-f117beda9c6d","Type":"ContainerDied","Data":"fdf3ff41a57be4339dc2d5dac351e18b733cb534b75cfedd7035e4c15fe7aa82"}
Oct 30 22:30:10.520408 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:10.520283    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-84a3cf-29889990-njn48"
Oct 30 22:30:10.520408 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:10.520385    5061 scope.go:122] "RemoveContainer" containerID="a47f1cbfe43da2bba66e716f3ab892bb0a148c73715c5b68b0f96ede014e8c39"
Oct 30 22:30:10.586430 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:10.586394    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-84a3cf-6c95488bd4-cjmzh" podUID="7e098fc6-5863-4aa7-a134-f70fc5380a0d" containerName="payload-processing" containerID="cri-o://37ba9b5cd8f7b071d28c86233af2bdb00bb896124c04340e88670ad1642596b3" gracePeriod=25
Oct 30 22:30:10.605062 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:10.605034    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cf662ac-1070-4ee6-bf5d-f117beda9c6d-maas-api-service-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:10.605062 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:10.605059    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45rsx\" (UniqueName: \"kubernetes.io/projected/7cf662ac-1070-4ee6-bf5d-f117beda9c6d-kube-api-access-45rsx\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:10.677441 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:10.677409    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-84a3cf-29889990-njn48"]
Oct 30 22:30:10.681284 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:10.681258    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-84a3cf-29889990-njn48"]
Oct 30 22:30:10.691408 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:10.691368    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-84a3cf-6cb65fb6cf-r64nq" podUID="63cdc87a-9959-41f0-b5c8-1df23cf0cda4" containerName="payload-pre-processing" containerID="cri-o://c28cfbfabc772b3ed0af12640598e35d422138fe420beef8bacf1a16879d1bcb" gracePeriod=25
Oct 30 22:30:12.236896 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:12.236853    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-84a3cf-ktgt6"]
Oct 30 22:30:12.237384 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:12.236919    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-84a3cf-ktgt6"]
Oct 30 22:30:12.528229 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:12.528147    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7cf662ac-1070-4ee6-bf5d-f117beda9c6d" path="/var/lib/kubelet/pods/7cf662ac-1070-4ee6-bf5d-f117beda9c6d/volumes"
Oct 30 22:30:12.528618 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:12.528601    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8441ac0b-2d54-4374-8407-1d3ac661fc3e" path="/var/lib/kubelet/pods/8441ac0b-2d54-4374-8407-1d3ac661fc3e/volumes"
Oct 30 22:30:15.091456 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:15.091416    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-84a3cf-6cb65fb6cf-r64nq" podUID="63cdc87a-9959-41f0-b5c8-1df23cf0cda4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:30:15.091888 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:15.091431    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-84a3cf-6c95488bd4-cjmzh" podUID="7e098fc6-5863-4aa7-a134-f70fc5380a0d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:30:15.521470 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:15.521392    5061 scope.go:122] "RemoveContainer" containerID="8872f3be6cc4c7a793c1a0c99d3acb82488ece9ae36963bbb2f30a3550aa0f46"
Oct 30 22:30:16.557329 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:16.557297    5061 generic.go:356] "Generic (PLEG): container finished" podID="d14224db-1f4f-45c3-a36b-8af75018a006" containerID="026f8958c182e76b3911a30ea02d45b27b76142cf4d79d3c4917809a95a60dee" exitCode=60
Oct 30 22:30:16.557799 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:16.557368    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5402dd-29889990-45272" event={"ID":"d14224db-1f4f-45c3-a36b-8af75018a006","Type":"ContainerDied","Data":"026f8958c182e76b3911a30ea02d45b27b76142cf4d79d3c4917809a95a60dee"}
Oct 30 22:30:16.557799 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:16.557421    5061 scope.go:122] "RemoveContainer" containerID="8872f3be6cc4c7a793c1a0c99d3acb82488ece9ae36963bbb2f30a3550aa0f46"
Oct 30 22:30:16.557799 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:16.557731    5061 scope.go:122] "RemoveContainer" containerID="026f8958c182e76b3911a30ea02d45b27b76142cf4d79d3c4917809a95a60dee"
Oct 30 22:30:16.557976 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:30:16.557957    5061 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-5402dd-29889990-45272_odh-ai-gateway-infra(d14224db-1f4f-45c3-a36b-8af75018a006)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5402dd-29889990-45272" podUID="d14224db-1f4f-45c3-a36b-8af75018a006"
Oct 30 22:30:17.521065 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:17.521036    5061 scope.go:122] "RemoveContainer" containerID="100e3948fac4a85a1a6d838aea832ef4564d6d90aa52c0ac74fa79a98339714a"
Oct 30 22:30:17.678344 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:17.678303    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5402dd-29889990-45272"]
Oct 30 22:30:17.837157 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:17.837134    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5402dd-29889990-45272"
Oct 30 22:30:17.995910 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:17.995871    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d14224db-1f4f-45c3-a36b-8af75018a006-kube-api-access-6jbkr\" (UniqueName: \"kubernetes.io/projected/d14224db-1f4f-45c3-a36b-8af75018a006-kube-api-access-6jbkr\") pod \"d14224db-1f4f-45c3-a36b-8af75018a006\" (UID: \"d14224db-1f4f-45c3-a36b-8af75018a006\") "
Oct 30 22:30:17.996089 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:17.995936    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d14224db-1f4f-45c3-a36b-8af75018a006-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d14224db-1f4f-45c3-a36b-8af75018a006-maas-api-service-ca\") pod \"d14224db-1f4f-45c3-a36b-8af75018a006\" (UID: \"d14224db-1f4f-45c3-a36b-8af75018a006\") "
Oct 30 22:30:17.996285 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:17.996260    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d14224db-1f4f-45c3-a36b-8af75018a006-maas-api-service-ca" pod "d14224db-1f4f-45c3-a36b-8af75018a006" (UID: "d14224db-1f4f-45c3-a36b-8af75018a006"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:17.996449 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:17.996434    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d14224db-1f4f-45c3-a36b-8af75018a006-maas-api-service-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:17.997940 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:17.997916    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d14224db-1f4f-45c3-a36b-8af75018a006-kube-api-access-6jbkr" pod "d14224db-1f4f-45c3-a36b-8af75018a006" (UID: "d14224db-1f4f-45c3-a36b-8af75018a006"). InnerVolumeSpecName "kube-api-access-6jbkr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:18.097285 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:18.097219    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jbkr\" (UniqueName: \"kubernetes.io/projected/d14224db-1f4f-45c3-a36b-8af75018a006-kube-api-access-6jbkr\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:18.521420 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:18.521389    5061 scope.go:122] "RemoveContainer" containerID="32bbac1f6f4207ab4251606419130d5dfa5fa39f6ca2bed518dcab46ddf66f7c"
Oct 30 22:30:18.572769 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:18.572738    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5402dd-29889990-45272"
Oct 30 22:30:18.572861 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:18.572769    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5402dd-29889990-45272" event={"ID":"d14224db-1f4f-45c3-a36b-8af75018a006","Type":"ContainerDied","Data":"0a0e3101419fb3302cb98f90bada528ef3bedc5e18cf1c63cd20b698ca63aae9"}
Oct 30 22:30:18.572861 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:18.572823    5061 scope.go:122] "RemoveContainer" containerID="026f8958c182e76b3911a30ea02d45b27b76142cf4d79d3c4917809a95a60dee"
Oct 30 22:30:18.578008 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:18.577977    5061 generic.go:356] "Generic (PLEG): container finished" podID="51015ddd-3122-4cfb-99bc-d4c041646c64" containerID="1ce959babd1d1072c1f65c5006a9623b9bce996e8fa67c141f0b8ee7278124c0" exitCode=60
Oct 30 22:30:18.578136 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:18.578025    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-4d8041-29889990-6zvnx" event={"ID":"51015ddd-3122-4cfb-99bc-d4c041646c64","Type":"ContainerDied","Data":"1ce959babd1d1072c1f65c5006a9623b9bce996e8fa67c141f0b8ee7278124c0"}
Oct 30 22:30:18.578558 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:18.578522    5061 scope.go:122] "RemoveContainer" containerID="1ce959babd1d1072c1f65c5006a9623b9bce996e8fa67c141f0b8ee7278124c0"
Oct 30 22:30:18.578962 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:30:18.578934    5061 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-4d8041-29889990-6zvnx_odh-ai-gateway-infra(51015ddd-3122-4cfb-99bc-d4c041646c64)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-4d8041-29889990-6zvnx" podUID="51015ddd-3122-4cfb-99bc-d4c041646c64"
Oct 30 22:30:18.587700 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:18.587675    5061 scope.go:122] "RemoveContainer" containerID="100e3948fac4a85a1a6d838aea832ef4564d6d90aa52c0ac74fa79a98339714a"
Oct 30 22:30:18.836319 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:18.836296    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5402dd-29889990-45272"]
Oct 30 22:30:18.838888 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:18.838863    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5402dd-29889990-45272"]
Oct 30 22:30:19.242105 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:19.242072    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-84a3cf-openshift-default-6b47cf679-n6j8d"]
Oct 30 22:30:19.242341 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:19.242307    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-84a3cf-openshift-default-6b47cf679-n6j8d" podUID="8cf03377-68fe-44d3-a000-3a4893ef6b8a" containerName="istio-proxy" containerID="cri-o://e5be667ce8b3fac749e9c6931a15edeb4f2b971358bd3cc24855c5f01ded15d9" gracePeriod=30
Oct 30 22:30:19.585332 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:19.585300    5061 generic.go:356] "Generic (PLEG): container finished" podID="333a34f7-9655-4ea6-9588-b9f5cb8fe305" containerID="24129dd37b62afc27f825c8c6e2e3d11abfdf6c310988bc74676eee9ac969da3" exitCode=60
Oct 30 22:30:19.585486 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:19.585372    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-02bd3b-29889990-77nwk" event={"ID":"333a34f7-9655-4ea6-9588-b9f5cb8fe305","Type":"ContainerDied","Data":"24129dd37b62afc27f825c8c6e2e3d11abfdf6c310988bc74676eee9ac969da3"}
Oct 30 22:30:19.585486 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:19.585438    5061 scope.go:122] "RemoveContainer" containerID="32bbac1f6f4207ab4251606419130d5dfa5fa39f6ca2bed518dcab46ddf66f7c"
Oct 30 22:30:19.585831 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:19.585811    5061 scope.go:122] "RemoveContainer" containerID="24129dd37b62afc27f825c8c6e2e3d11abfdf6c310988bc74676eee9ac969da3"
Oct 30 22:30:19.586118 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:30:19.586094    5061 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-02bd3b-29889990-77nwk_odh-ai-gateway-infra(333a34f7-9655-4ea6-9588-b9f5cb8fe305)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-02bd3b-29889990-77nwk" podUID="333a34f7-9655-4ea6-9588-b9f5cb8fe305"
Oct 30 22:30:19.740670 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:19.740636    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-4d8041-29889990-6zvnx"]
Oct 30 22:30:19.888108 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:19.888085    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-4d8041-29889990-6zvnx"
Oct 30 22:30:19.914237 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:19.914209    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51015ddd-3122-4cfb-99bc-d4c041646c64-kube-api-access-s2jl4\" (UniqueName: \"kubernetes.io/projected/51015ddd-3122-4cfb-99bc-d4c041646c64-kube-api-access-s2jl4\") pod \"51015ddd-3122-4cfb-99bc-d4c041646c64\" (UID: \"51015ddd-3122-4cfb-99bc-d4c041646c64\") "
Oct 30 22:30:19.914383 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:19.914335    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51015ddd-3122-4cfb-99bc-d4c041646c64-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51015ddd-3122-4cfb-99bc-d4c041646c64-maas-api-service-ca\") pod \"51015ddd-3122-4cfb-99bc-d4c041646c64\" (UID: \"51015ddd-3122-4cfb-99bc-d4c041646c64\") "
Oct 30 22:30:19.914683 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:19.914661    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51015ddd-3122-4cfb-99bc-d4c041646c64-maas-api-service-ca" pod "51015ddd-3122-4cfb-99bc-d4c041646c64" (UID: "51015ddd-3122-4cfb-99bc-d4c041646c64"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:19.916222 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:19.916194    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51015ddd-3122-4cfb-99bc-d4c041646c64-kube-api-access-s2jl4" pod "51015ddd-3122-4cfb-99bc-d4c041646c64" (UID: "51015ddd-3122-4cfb-99bc-d4c041646c64"). InnerVolumeSpecName "kube-api-access-s2jl4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:20.015633 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:20.015603    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51015ddd-3122-4cfb-99bc-d4c041646c64-maas-api-service-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:20.015633 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:20.015630    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2jl4\" (UniqueName: \"kubernetes.io/projected/51015ddd-3122-4cfb-99bc-d4c041646c64-kube-api-access-s2jl4\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:20.527049 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:20.527014    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d14224db-1f4f-45c3-a36b-8af75018a006" path="/var/lib/kubelet/pods/d14224db-1f4f-45c3-a36b-8af75018a006/volumes"
Oct 30 22:30:20.596885 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:20.596852    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-4d8041-29889990-6zvnx"
Oct 30 22:30:20.597030 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:20.596880    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-4d8041-29889990-6zvnx" event={"ID":"51015ddd-3122-4cfb-99bc-d4c041646c64","Type":"ContainerDied","Data":"7b6fa411593fb1f3d4c4b20ff12e5a89f797bc88f1dd9780e7e23f65ce12e89d"}
Oct 30 22:30:20.597030 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:20.596938    5061 scope.go:122] "RemoveContainer" containerID="1ce959babd1d1072c1f65c5006a9623b9bce996e8fa67c141f0b8ee7278124c0"
Oct 30 22:30:20.612718 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:20.612688    5061 patch_prober.go:29] interesting pod/e2e-inf-b-84a3cf-openshift-default-6b47cf679-n6j8d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:30:20.612825 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:20.612740    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-84a3cf-openshift-default-6b47cf679-n6j8d" podUID="8cf03377-68fe-44d3-a000-3a4893ef6b8a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:30:20.705214 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:20.705185    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-02bd3b-29889990-77nwk"]
Oct 30 22:30:20.753293 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:20.753255    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-4d8041-29889990-6zvnx"]
Oct 30 22:30:20.759586 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:20.759556    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-4d8041-29889990-6zvnx"]
Oct 30 22:30:20.863004 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:20.862976    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-02bd3b-29889990-77nwk"
Oct 30 22:30:20.922903 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:20.922864    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/333a34f7-9655-4ea6-9588-b9f5cb8fe305-kube-api-access-vrrql\" (UniqueName: \"kubernetes.io/projected/333a34f7-9655-4ea6-9588-b9f5cb8fe305-kube-api-access-vrrql\") pod \"333a34f7-9655-4ea6-9588-b9f5cb8fe305\" (UID: \"333a34f7-9655-4ea6-9588-b9f5cb8fe305\") "
Oct 30 22:30:20.923400 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:20.923034    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/333a34f7-9655-4ea6-9588-b9f5cb8fe305-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/333a34f7-9655-4ea6-9588-b9f5cb8fe305-maas-api-service-ca\") pod \"333a34f7-9655-4ea6-9588-b9f5cb8fe305\" (UID: \"333a34f7-9655-4ea6-9588-b9f5cb8fe305\") "
Oct 30 22:30:20.923400 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:20.923379    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/333a34f7-9655-4ea6-9588-b9f5cb8fe305-maas-api-service-ca" pod "333a34f7-9655-4ea6-9588-b9f5cb8fe305" (UID: "333a34f7-9655-4ea6-9588-b9f5cb8fe305"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:20.925025 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:20.925006    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/333a34f7-9655-4ea6-9588-b9f5cb8fe305-kube-api-access-vrrql" pod "333a34f7-9655-4ea6-9588-b9f5cb8fe305" (UID: "333a34f7-9655-4ea6-9588-b9f5cb8fe305"). InnerVolumeSpecName "kube-api-access-vrrql". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:21.024047 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:21.024012    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/333a34f7-9655-4ea6-9588-b9f5cb8fe305-maas-api-service-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:21.024047 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:21.024043    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrrql\" (UniqueName: \"kubernetes.io/projected/333a34f7-9655-4ea6-9588-b9f5cb8fe305-kube-api-access-vrrql\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:21.603946 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:21.603913    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-02bd3b-29889990-77nwk"
Oct 30 22:30:21.604156 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:21.603946    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-02bd3b-29889990-77nwk" event={"ID":"333a34f7-9655-4ea6-9588-b9f5cb8fe305","Type":"ContainerDied","Data":"65571dbe669318b99a6c4f6998f53baf78600abb5b38bddbfacb147f8912f6ad"}
Oct 30 22:30:21.604156 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:21.604005    5061 scope.go:122] "RemoveContainer" containerID="24129dd37b62afc27f825c8c6e2e3d11abfdf6c310988bc74676eee9ac969da3"
Oct 30 22:30:21.758241 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:21.758207    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-02bd3b-29889990-77nwk"]
Oct 30 22:30:21.805688 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:21.805655    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-02bd3b-29889990-77nwk"]
Oct 30 22:30:22.528128 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:22.528095    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="333a34f7-9655-4ea6-9588-b9f5cb8fe305" path="/var/lib/kubelet/pods/333a34f7-9655-4ea6-9588-b9f5cb8fe305/volumes"
Oct 30 22:30:22.528518 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:22.528504    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51015ddd-3122-4cfb-99bc-d4c041646c64" path="/var/lib/kubelet/pods/51015ddd-3122-4cfb-99bc-d4c041646c64/volumes"
Oct 30 22:30:24.502359 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:24.502335    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-84a3cf-openshift-default-6b47cf679-n6j8d"
Oct 30 22:30:24.561978 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:24.561950    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8cf03377-68fe-44d3-a000-3a4893ef6b8a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cf03377-68fe-44d3-a000-3a4893ef6b8a-credential-socket\") pod \"8cf03377-68fe-44d3-a000-3a4893ef6b8a\" (UID: \"8cf03377-68fe-44d3-a000-3a4893ef6b8a\") "
Oct 30 22:30:24.562122 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:24.561991    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8cf03377-68fe-44d3-a000-3a4893ef6b8a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8cf03377-68fe-44d3-a000-3a4893ef6b8a-istio-data\") pod \"8cf03377-68fe-44d3-a000-3a4893ef6b8a\" (UID: \"8cf03377-68fe-44d3-a000-3a4893ef6b8a\") "
Oct 30 22:30:24.562122 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:24.562042    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8cf03377-68fe-44d3-a000-3a4893ef6b8a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8cf03377-68fe-44d3-a000-3a4893ef6b8a-workload-certs\") pod \"8cf03377-68fe-44d3-a000-3a4893ef6b8a\" (UID: \"8cf03377-68fe-44d3-a000-3a4893ef6b8a\") "
Oct 30 22:30:24.562122 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:24.562062    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8cf03377-68fe-44d3-a000-3a4893ef6b8a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8cf03377-68fe-44d3-a000-3a4893ef6b8a-istio-podinfo\") pod \"8cf03377-68fe-44d3-a000-3a4893ef6b8a\" (UID: \"8cf03377-68fe-44d3-a000-3a4893ef6b8a\") "
Oct 30 22:30:24.562122 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:24.562083    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8cf03377-68fe-44d3-a000-3a4893ef6b8a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cf03377-68fe-44d3-a000-3a4893ef6b8a-workload-socket\") pod \"8cf03377-68fe-44d3-a000-3a4893ef6b8a\" (UID: \"8cf03377-68fe-44d3-a000-3a4893ef6b8a\") "
Oct 30 22:30:24.562333 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:24.562133    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8cf03377-68fe-44d3-a000-3a4893ef6b8a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8cf03377-68fe-44d3-a000-3a4893ef6b8a-istiod-ca-cert\") pod \"8cf03377-68fe-44d3-a000-3a4893ef6b8a\" (UID: \"8cf03377-68fe-44d3-a000-3a4893ef6b8a\") "
Oct 30 22:30:24.562333 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:24.562229    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cf03377-68fe-44d3-a000-3a4893ef6b8a-credential-socket" pod "8cf03377-68fe-44d3-a000-3a4893ef6b8a" (UID: "8cf03377-68fe-44d3-a000-3a4893ef6b8a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:24.562447 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:24.562353    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cf03377-68fe-44d3-a000-3a4893ef6b8a-workload-certs" pod "8cf03377-68fe-44d3-a000-3a4893ef6b8a" (UID: "8cf03377-68fe-44d3-a000-3a4893ef6b8a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:24.562447 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:24.562377    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cf03377-68fe-44d3-a000-3a4893ef6b8a-kube-api-access-fqx7z\" (UniqueName: \"kubernetes.io/projected/8cf03377-68fe-44d3-a000-3a4893ef6b8a-kube-api-access-fqx7z\") pod \"8cf03377-68fe-44d3-a000-3a4893ef6b8a\" (UID: \"8cf03377-68fe-44d3-a000-3a4893ef6b8a\") "
Oct 30 22:30:24.562447 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:24.562393    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cf03377-68fe-44d3-a000-3a4893ef6b8a-istio-data" pod "8cf03377-68fe-44d3-a000-3a4893ef6b8a" (UID: "8cf03377-68fe-44d3-a000-3a4893ef6b8a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:24.562447 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:24.562430    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cf03377-68fe-44d3-a000-3a4893ef6b8a-workload-socket" pod "8cf03377-68fe-44d3-a000-3a4893ef6b8a" (UID: "8cf03377-68fe-44d3-a000-3a4893ef6b8a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:24.562702 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:24.562448    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cf03377-68fe-44d3-a000-3a4893ef6b8a-istio-token\" (UniqueName: \"kubernetes.io/projected/8cf03377-68fe-44d3-a000-3a4893ef6b8a-istio-token\") pod \"8cf03377-68fe-44d3-a000-3a4893ef6b8a\" (UID: \"8cf03377-68fe-44d3-a000-3a4893ef6b8a\") "
Oct 30 22:30:24.562702 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:24.562515    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8cf03377-68fe-44d3-a000-3a4893ef6b8a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8cf03377-68fe-44d3-a000-3a4893ef6b8a-istio-envoy\") pod \"8cf03377-68fe-44d3-a000-3a4893ef6b8a\" (UID: \"8cf03377-68fe-44d3-a000-3a4893ef6b8a\") "
Oct 30 22:30:24.562702 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:24.562560    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cf03377-68fe-44d3-a000-3a4893ef6b8a-istiod-ca-cert" pod "8cf03377-68fe-44d3-a000-3a4893ef6b8a" (UID: "8cf03377-68fe-44d3-a000-3a4893ef6b8a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:24.563026 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:24.563002    5061 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cf03377-68fe-44d3-a000-3a4893ef6b8a-credential-socket\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:24.563194 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:24.563169    5061 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8cf03377-68fe-44d3-a000-3a4893ef6b8a-istio-data\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:24.563302 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:24.563198    5061 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8cf03377-68fe-44d3-a000-3a4893ef6b8a-workload-certs\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:24.563302 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:24.563214    5061 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cf03377-68fe-44d3-a000-3a4893ef6b8a-workload-socket\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:24.563302 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:24.563227    5061 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8cf03377-68fe-44d3-a000-3a4893ef6b8a-istiod-ca-cert\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:24.564722 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:24.564693    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cf03377-68fe-44d3-a000-3a4893ef6b8a-kube-api-access-fqx7z" pod "8cf03377-68fe-44d3-a000-3a4893ef6b8a" (UID: "8cf03377-68fe-44d3-a000-3a4893ef6b8a"). InnerVolumeSpecName "kube-api-access-fqx7z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:24.564722 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:24.564705    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cf03377-68fe-44d3-a000-3a4893ef6b8a-istio-envoy" pod "8cf03377-68fe-44d3-a000-3a4893ef6b8a" (UID: "8cf03377-68fe-44d3-a000-3a4893ef6b8a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:24.564847 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:24.564690    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8cf03377-68fe-44d3-a000-3a4893ef6b8a-istio-podinfo" pod "8cf03377-68fe-44d3-a000-3a4893ef6b8a" (UID: "8cf03377-68fe-44d3-a000-3a4893ef6b8a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:30:24.564847 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:24.564727    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cf03377-68fe-44d3-a000-3a4893ef6b8a-istio-token" pod "8cf03377-68fe-44d3-a000-3a4893ef6b8a" (UID: "8cf03377-68fe-44d3-a000-3a4893ef6b8a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:24.625810 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:24.625784    5061 generic.go:356] "Generic (PLEG): container finished" podID="8cf03377-68fe-44d3-a000-3a4893ef6b8a" containerID="e5be667ce8b3fac749e9c6931a15edeb4f2b971358bd3cc24855c5f01ded15d9" exitCode=0
Oct 30 22:30:24.625920 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:24.625835    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-84a3cf-openshift-default-6b47cf679-n6j8d"
Oct 30 22:30:24.625920 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:24.625866    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-84a3cf-openshift-default-6b47cf679-n6j8d" event={"ID":"8cf03377-68fe-44d3-a000-3a4893ef6b8a","Type":"ContainerDied","Data":"e5be667ce8b3fac749e9c6931a15edeb4f2b971358bd3cc24855c5f01ded15d9"}
Oct 30 22:30:24.626009 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:24.625920    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-84a3cf-openshift-default-6b47cf679-n6j8d" event={"ID":"8cf03377-68fe-44d3-a000-3a4893ef6b8a","Type":"ContainerDied","Data":"2e79d8da5317b01f9d36432d61de4883da77d24b5534c945073b87333d21596a"}
Oct 30 22:30:24.626009 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:24.625942    5061 scope.go:122] "RemoveContainer" containerID="e5be667ce8b3fac749e9c6931a15edeb4f2b971358bd3cc24855c5f01ded15d9"
Oct 30 22:30:24.635742 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:24.635727    5061 scope.go:122] "RemoveContainer" containerID="3b1e4f5beb386b9fd37021e9ec9fb3dc409435a41568715df942b5d8c5e1b869"
Oct 30 22:30:24.645474 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:24.645454    5061 scope.go:122] "RemoveContainer" containerID="e5be667ce8b3fac749e9c6931a15edeb4f2b971358bd3cc24855c5f01ded15d9"
Oct 30 22:30:24.645743 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:30:24.645724    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5be667ce8b3fac749e9c6931a15edeb4f2b971358bd3cc24855c5f01ded15d9\": container with ID starting with e5be667ce8b3fac749e9c6931a15edeb4f2b971358bd3cc24855c5f01ded15d9 not found: ID does not exist" containerID="e5be667ce8b3fac749e9c6931a15edeb4f2b971358bd3cc24855c5f01ded15d9"
Oct 30 22:30:24.645820 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:24.645752    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5be667ce8b3fac749e9c6931a15edeb4f2b971358bd3cc24855c5f01ded15d9"} err="failed to get container status \"e5be667ce8b3fac749e9c6931a15edeb4f2b971358bd3cc24855c5f01ded15d9\": rpc error: code = NotFound desc = could not find container \"e5be667ce8b3fac749e9c6931a15edeb4f2b971358bd3cc24855c5f01ded15d9\": container with ID starting with e5be667ce8b3fac749e9c6931a15edeb4f2b971358bd3cc24855c5f01ded15d9 not found: ID does not exist"
Oct 30 22:30:24.645820 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:24.645769    5061 scope.go:122] "RemoveContainer" containerID="3b1e4f5beb386b9fd37021e9ec9fb3dc409435a41568715df942b5d8c5e1b869"
Oct 30 22:30:24.645984 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:30:24.645969    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b1e4f5beb386b9fd37021e9ec9fb3dc409435a41568715df942b5d8c5e1b869\": container with ID starting with 3b1e4f5beb386b9fd37021e9ec9fb3dc409435a41568715df942b5d8c5e1b869 not found: ID does not exist" containerID="3b1e4f5beb386b9fd37021e9ec9fb3dc409435a41568715df942b5d8c5e1b869"
Oct 30 22:30:24.646046 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:24.645987    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b1e4f5beb386b9fd37021e9ec9fb3dc409435a41568715df942b5d8c5e1b869"} err="failed to get container status \"3b1e4f5beb386b9fd37021e9ec9fb3dc409435a41568715df942b5d8c5e1b869\": rpc error: code = NotFound desc = could not find container \"3b1e4f5beb386b9fd37021e9ec9fb3dc409435a41568715df942b5d8c5e1b869\": container with ID starting with 3b1e4f5beb386b9fd37021e9ec9fb3dc409435a41568715df942b5d8c5e1b869 not found: ID does not exist"
Oct 30 22:30:24.664412 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:24.664384    5061 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8cf03377-68fe-44d3-a000-3a4893ef6b8a-istio-podinfo\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:24.664412 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:24.664413    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqx7z\" (UniqueName: \"kubernetes.io/projected/8cf03377-68fe-44d3-a000-3a4893ef6b8a-kube-api-access-fqx7z\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:24.664582 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:24.664425    5061 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8cf03377-68fe-44d3-a000-3a4893ef6b8a-istio-token\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:24.664582 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:24.664433    5061 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8cf03377-68fe-44d3-a000-3a4893ef6b8a-istio-envoy\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:24.666388 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:30:24.666364    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 22:30:24.769631 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:24.769598    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-84a3cf-openshift-default-6b47cf679-n6j8d"]
Oct 30 22:30:24.775481 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:24.775455    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-84a3cf-openshift-default-6b47cf679-n6j8d"]
Oct 30 22:30:25.091010 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:25.090977    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-84a3cf-6c95488bd4-cjmzh" podUID="7e098fc6-5863-4aa7-a134-f70fc5380a0d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:30:25.091010 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:25.090994    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-84a3cf-6cb65fb6cf-r64nq" podUID="63cdc87a-9959-41f0-b5c8-1df23cf0cda4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:30:26.527809 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:26.527777    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8cf03377-68fe-44d3-a000-3a4893ef6b8a" path="/var/lib/kubelet/pods/8cf03377-68fe-44d3-a000-3a4893ef6b8a/volumes"
Oct 30 22:30:29.278566 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:29.278512    5061 scope.go:122] "RemoveContainer" containerID="e617e63effe6985b880ad7e2ec66b5e5ee7fc768999f0d7945131ccdfa69f94e"
Oct 30 22:30:29.287819 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:29.287795    5061 scope.go:122] "RemoveContainer" containerID="acab0bdf919b22b07ef8c5dfc098f5d77577e07e0bd0e2c447feef0616d07b8a"
Oct 30 22:30:29.300402 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:29.300381    5061 scope.go:122] "RemoveContainer" containerID="bb41e5567f2211e56326899b72524c6b58b640f09f9842461f9af2133fdb048a"
Oct 30 22:30:29.308351 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:29.308331    5061 scope.go:122] "RemoveContainer" containerID="8c86c3de310ba46ff80d5698622bcf654dd7deae2dc7a0a09907b50dd59d6e06"
Oct 30 22:30:29.316643 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:29.316624    5061 scope.go:122] "RemoveContainer" containerID="f6e138f9074b6788e957dff55a8390451591198a10f4fc7e67bbbeb7a820d27f"
Oct 30 22:30:29.324412 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:29.324390    5061 scope.go:122] "RemoveContainer" containerID="28acd13885beb0938d1966b3106ad9025be66edda54a2a935c198c23d8d34f02"
Oct 30 22:30:30.045666 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:30.045634    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-4d8041-r6xgc"]
Oct 30 22:30:30.046120 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:30.046107    5061 cpu_manager.go:414] "containerMap: removing container" podUID="51015ddd-3122-4cfb-99bc-d4c041646c64" containerName="cleanup"
Oct 30 22:30:30.046189 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:30.046121    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51015ddd-3122-4cfb-99bc-d4c041646c64" containerName="cleanup"
Oct 30 22:30:30.046189 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:30.046132    5061 cpu_manager.go:414] "containerMap: removing container" podUID="d14224db-1f4f-45c3-a36b-8af75018a006" containerName="cleanup"
Oct 30 22:30:30.046189 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:30.046137    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d14224db-1f4f-45c3-a36b-8af75018a006" containerName="cleanup"
Oct 30 22:30:30.046189 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:30.046162    5061 cpu_manager.go:414] "containerMap: removing container" podUID="d14224db-1f4f-45c3-a36b-8af75018a006" containerName="cleanup"
Oct 30 22:30:30.046189 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:30.046168    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d14224db-1f4f-45c3-a36b-8af75018a006" containerName="cleanup"
Oct 30 22:30:30.046189 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:30.046180    5061 cpu_manager.go:414] "containerMap: removing container" podUID="333a34f7-9655-4ea6-9588-b9f5cb8fe305" containerName="cleanup"
Oct 30 22:30:30.046189 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:30.046186    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="333a34f7-9655-4ea6-9588-b9f5cb8fe305" containerName="cleanup"
Oct 30 22:30:30.046409 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:30.046193    5061 cpu_manager.go:414] "containerMap: removing container" podUID="8cf03377-68fe-44d3-a000-3a4893ef6b8a" containerName="istio-proxy"
Oct 30 22:30:30.046409 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:30.046198    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cf03377-68fe-44d3-a000-3a4893ef6b8a" containerName="istio-proxy"
Oct 30 22:30:30.046409 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:30.046206    5061 cpu_manager.go:414] "containerMap: removing container" podUID="333a34f7-9655-4ea6-9588-b9f5cb8fe305" containerName="cleanup"
Oct 30 22:30:30.046409 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:30.046211    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="333a34f7-9655-4ea6-9588-b9f5cb8fe305" containerName="cleanup"
Oct 30 22:30:30.046409 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:30.046223    5061 cpu_manager.go:414] "containerMap: removing container" podUID="8cf03377-68fe-44d3-a000-3a4893ef6b8a" containerName="istio-proxy"
Oct 30 22:30:30.046409 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:30.046231    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cf03377-68fe-44d3-a000-3a4893ef6b8a" containerName="istio-proxy"
Oct 30 22:30:30.046409 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:30.046245    5061 cpu_manager.go:414] "containerMap: removing container" podUID="7cf662ac-1070-4ee6-bf5d-f117beda9c6d" containerName="cleanup"
Oct 30 22:30:30.046409 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:30.046253    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cf662ac-1070-4ee6-bf5d-f117beda9c6d" containerName="cleanup"
Oct 30 22:30:30.046409 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:30.046264    5061 cpu_manager.go:414] "containerMap: removing container" podUID="7cf662ac-1070-4ee6-bf5d-f117beda9c6d" containerName="cleanup"
Oct 30 22:30:30.046409 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:30.046270    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cf662ac-1070-4ee6-bf5d-f117beda9c6d" containerName="cleanup"
Oct 30 22:30:30.046409 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:30.046293    5061 cpu_manager.go:414] "containerMap: removing container" podUID="51015ddd-3122-4cfb-99bc-d4c041646c64" containerName="cleanup"
Oct 30 22:30:30.046409 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:30.046297    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51015ddd-3122-4cfb-99bc-d4c041646c64" containerName="cleanup"
Oct 30 22:30:30.046409 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:30.046377    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="51015ddd-3122-4cfb-99bc-d4c041646c64" containerName="cleanup"
Oct 30 22:30:30.046409 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:30.046385    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="d14224db-1f4f-45c3-a36b-8af75018a006" containerName="cleanup"
Oct 30 22:30:30.046409 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:30.046393    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="51015ddd-3122-4cfb-99bc-d4c041646c64" containerName="cleanup"
Oct 30 22:30:30.046409 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:30.046399    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="d14224db-1f4f-45c3-a36b-8af75018a006" containerName="cleanup"
Oct 30 22:30:30.046409 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:30.046404    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="d14224db-1f4f-45c3-a36b-8af75018a006" containerName="cleanup"
Oct 30 22:30:30.046409 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:30.046411    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="333a34f7-9655-4ea6-9588-b9f5cb8fe305" containerName="cleanup"
Oct 30 22:30:30.046409 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:30.046416    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="51015ddd-3122-4cfb-99bc-d4c041646c64" containerName="cleanup"
Oct 30 22:30:30.047022 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:30.046421    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cf03377-68fe-44d3-a000-3a4893ef6b8a" containerName="istio-proxy"
Oct 30 22:30:30.047022 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:30.046426    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="333a34f7-9655-4ea6-9588-b9f5cb8fe305" containerName="cleanup"
Oct 30 22:30:30.047022 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:30.046433    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cf662ac-1070-4ee6-bf5d-f117beda9c6d" containerName="cleanup"
Oct 30 22:30:30.047022 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:30.046439    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cf662ac-1070-4ee6-bf5d-f117beda9c6d" containerName="cleanup"
Oct 30 22:30:30.047022 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:30.046446    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="333a34f7-9655-4ea6-9588-b9f5cb8fe305" containerName="cleanup"
Oct 30 22:30:30.051516 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:30.051492    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-4d8041-r6xgc"
Oct 30 22:30:30.101283 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:30.101244    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:30:30.118065 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:30.118027    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1711e053-6008-4125-8479-502130dcecae-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-4d8041-r6xgc\" (UID: \"1711e053-6008-4125-8479-502130dcecae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-4d8041-r6xgc"
Oct 30 22:30:30.118255 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:30.118092    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24mw9\" (UniqueName: \"kubernetes.io/projected/1711e053-6008-4125-8479-502130dcecae-kube-api-access-24mw9\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-4d8041-r6xgc\" (UID: \"1711e053-6008-4125-8479-502130dcecae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-4d8041-r6xgc"
Oct 30 22:30:30.118255 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:30.118170    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1711e053-6008-4125-8479-502130dcecae-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-4d8041-r6xgc\" (UID: \"1711e053-6008-4125-8479-502130dcecae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-4d8041-r6xgc"
Oct 30 22:30:30.148142 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:30.148109    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:30:30.148331 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:30.148263    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gdmfg\""
Oct 30 22:30:30.148331 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:30.148264    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:30:30.150937 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:30.150911    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-4d8041-r6xgc"]
Oct 30 22:30:30.219373 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:30.219340    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1711e053-6008-4125-8479-502130dcecae-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-4d8041-r6xgc\" (UID: \"1711e053-6008-4125-8479-502130dcecae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-4d8041-r6xgc"
Oct 30 22:30:30.219587 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:30.219554    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1711e053-6008-4125-8479-502130dcecae-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-4d8041-r6xgc\" (UID: \"1711e053-6008-4125-8479-502130dcecae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-4d8041-r6xgc"
Oct 30 22:30:30.219755 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:30.219609    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24mw9\" (UniqueName: \"kubernetes.io/projected/1711e053-6008-4125-8479-502130dcecae-kube-api-access-24mw9\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-4d8041-r6xgc\" (UID: \"1711e053-6008-4125-8479-502130dcecae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-4d8041-r6xgc"
Oct 30 22:30:30.220159 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:30.220139    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1711e053-6008-4125-8479-502130dcecae-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-4d8041-r6xgc\" (UID: \"1711e053-6008-4125-8479-502130dcecae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-4d8041-r6xgc"
Oct 30 22:30:30.221826 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:30.221804    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1711e053-6008-4125-8479-502130dcecae-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-4d8041-r6xgc\" (UID: \"1711e053-6008-4125-8479-502130dcecae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-4d8041-r6xgc"
Oct 30 22:30:30.273868 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:30.273835    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24mw9\" (UniqueName: \"kubernetes.io/projected/1711e053-6008-4125-8479-502130dcecae-kube-api-access-24mw9\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-4d8041-r6xgc\" (UID: \"1711e053-6008-4125-8479-502130dcecae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-4d8041-r6xgc"
Oct 30 22:30:30.365001 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:30.364919    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-4d8041-r6xgc"
Oct 30 22:30:30.736965 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:30.735931    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-4d8041-r6xgc"]
Oct 30 22:30:30.760660 ip-10-0-130-60 kubenswrapper[5061]: W1030 22:30:30.760628    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1711e053_6008_4125_8479_502130dcecae.slice/crio-f81359317c1be0abb4591c6267e755adfa5b732696717ca88a8391a96fb8953b WatchSource:0}: Error finding container f81359317c1be0abb4591c6267e755adfa5b732696717ca88a8391a96fb8953b: Status 404 returned error can't find the container with id f81359317c1be0abb4591c6267e755adfa5b732696717ca88a8391a96fb8953b
Oct 30 22:30:30.846825 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:30.846798    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-84a3cf-6c95488bd4-cjmzh"
Oct 30 22:30:30.932850 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:30.932819    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-84a3cf-6cb65fb6cf-r64nq"
Oct 30 22:30:30.949350 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:30.949295    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e098fc6-5863-4aa7-a134-f70fc5380a0d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e098fc6-5863-4aa7-a134-f70fc5380a0d-plugins-config-volume\") pod \"7e098fc6-5863-4aa7-a134-f70fc5380a0d\" (UID: \"7e098fc6-5863-4aa7-a134-f70fc5380a0d\") "
Oct 30 22:30:30.949470 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:30.949458    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e098fc6-5863-4aa7-a134-f70fc5380a0d-kube-api-access-9bzcs\" (UniqueName: \"kubernetes.io/projected/7e098fc6-5863-4aa7-a134-f70fc5380a0d-kube-api-access-9bzcs\") pod \"7e098fc6-5863-4aa7-a134-f70fc5380a0d\" (UID: \"7e098fc6-5863-4aa7-a134-f70fc5380a0d\") "
Oct 30 22:30:30.949739 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:30.949713    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e098fc6-5863-4aa7-a134-f70fc5380a0d-plugins-config-volume" pod "7e098fc6-5863-4aa7-a134-f70fc5380a0d" (UID: "7e098fc6-5863-4aa7-a134-f70fc5380a0d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:30.951384 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:30.951367    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e098fc6-5863-4aa7-a134-f70fc5380a0d-kube-api-access-9bzcs" pod "7e098fc6-5863-4aa7-a134-f70fc5380a0d" (UID: "7e098fc6-5863-4aa7-a134-f70fc5380a0d"). InnerVolumeSpecName "kube-api-access-9bzcs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:31.050595 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:31.050559    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63cdc87a-9959-41f0-b5c8-1df23cf0cda4-kube-api-access-4dnvm\" (UniqueName: \"kubernetes.io/projected/63cdc87a-9959-41f0-b5c8-1df23cf0cda4-kube-api-access-4dnvm\") pod \"63cdc87a-9959-41f0-b5c8-1df23cf0cda4\" (UID: \"63cdc87a-9959-41f0-b5c8-1df23cf0cda4\") "
Oct 30 22:30:31.050783 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:31.050604    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63cdc87a-9959-41f0-b5c8-1df23cf0cda4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63cdc87a-9959-41f0-b5c8-1df23cf0cda4-plugins-config-volume\") pod \"63cdc87a-9959-41f0-b5c8-1df23cf0cda4\" (UID: \"63cdc87a-9959-41f0-b5c8-1df23cf0cda4\") "
Oct 30 22:30:31.050869 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:31.050860    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bzcs\" (UniqueName: \"kubernetes.io/projected/7e098fc6-5863-4aa7-a134-f70fc5380a0d-kube-api-access-9bzcs\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:31.050940 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:31.050875    5061 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e098fc6-5863-4aa7-a134-f70fc5380a0d-plugins-config-volume\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:31.051055 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:31.051030    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63cdc87a-9959-41f0-b5c8-1df23cf0cda4-plugins-config-volume" pod "63cdc87a-9959-41f0-b5c8-1df23cf0cda4" (UID: "63cdc87a-9959-41f0-b5c8-1df23cf0cda4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:31.052734 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:31.052711    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63cdc87a-9959-41f0-b5c8-1df23cf0cda4-kube-api-access-4dnvm" pod "63cdc87a-9959-41f0-b5c8-1df23cf0cda4" (UID: "63cdc87a-9959-41f0-b5c8-1df23cf0cda4"). InnerVolumeSpecName "kube-api-access-4dnvm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:31.152465 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:31.152432    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dnvm\" (UniqueName: \"kubernetes.io/projected/63cdc87a-9959-41f0-b5c8-1df23cf0cda4-kube-api-access-4dnvm\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:31.152465 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:31.152461    5061 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63cdc87a-9959-41f0-b5c8-1df23cf0cda4-plugins-config-volume\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:31.670947 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:31.670905    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-4d8041-r6xgc" event={"ID":"1711e053-6008-4125-8479-502130dcecae","Type":"ContainerStarted","Data":"2affb77087b92c8c42090ae65d4419ef2ab4e1c187fcd8c22244217cb1bf9fd5"}
Oct 30 22:30:31.671457 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:31.670970    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-4d8041-r6xgc" event={"ID":"1711e053-6008-4125-8479-502130dcecae","Type":"ContainerStarted","Data":"f81359317c1be0abb4591c6267e755adfa5b732696717ca88a8391a96fb8953b"}
Oct 30 22:30:31.672204 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:31.672181    5061 generic.go:356] "Generic (PLEG): container finished" podID="63cdc87a-9959-41f0-b5c8-1df23cf0cda4" containerID="c28cfbfabc772b3ed0af12640598e35d422138fe420beef8bacf1a16879d1bcb" exitCode=0
Oct 30 22:30:31.672282 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:31.672215    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-84a3cf-6cb65fb6cf-r64nq"
Oct 30 22:30:31.672282 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:31.672255    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-84a3cf-6cb65fb6cf-r64nq" event={"ID":"63cdc87a-9959-41f0-b5c8-1df23cf0cda4","Type":"ContainerDied","Data":"c28cfbfabc772b3ed0af12640598e35d422138fe420beef8bacf1a16879d1bcb"}
Oct 30 22:30:31.672379 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:31.672305    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-84a3cf-6cb65fb6cf-r64nq" event={"ID":"63cdc87a-9959-41f0-b5c8-1df23cf0cda4","Type":"ContainerDied","Data":"1fe812826553827988493bc7739b24f93a49da3c344e6754c78022c668688d64"}
Oct 30 22:30:31.672379 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:31.672334    5061 scope.go:122] "RemoveContainer" containerID="c28cfbfabc772b3ed0af12640598e35d422138fe420beef8bacf1a16879d1bcb"
Oct 30 22:30:31.673685 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:31.673487    5061 generic.go:356] "Generic (PLEG): container finished" podID="7e098fc6-5863-4aa7-a134-f70fc5380a0d" containerID="37ba9b5cd8f7b071d28c86233af2bdb00bb896124c04340e88670ad1642596b3" exitCode=0
Oct 30 22:30:31.673685 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:31.673555    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-84a3cf-6c95488bd4-cjmzh"
Oct 30 22:30:31.673685 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:31.673557    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-84a3cf-6c95488bd4-cjmzh" event={"ID":"7e098fc6-5863-4aa7-a134-f70fc5380a0d","Type":"ContainerDied","Data":"37ba9b5cd8f7b071d28c86233af2bdb00bb896124c04340e88670ad1642596b3"}
Oct 30 22:30:31.673685 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:31.673672    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-84a3cf-6c95488bd4-cjmzh" event={"ID":"7e098fc6-5863-4aa7-a134-f70fc5380a0d","Type":"ContainerDied","Data":"3cfd0325bd8c88c7e29e4d19c5fdf4578861a91b5a5049ec9d680babe24487cc"}
Oct 30 22:30:31.682965 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:31.682871    5061 scope.go:122] "RemoveContainer" containerID="c28cfbfabc772b3ed0af12640598e35d422138fe420beef8bacf1a16879d1bcb"
Oct 30 22:30:31.683263 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:30:31.683150    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c28cfbfabc772b3ed0af12640598e35d422138fe420beef8bacf1a16879d1bcb\": container with ID starting with c28cfbfabc772b3ed0af12640598e35d422138fe420beef8bacf1a16879d1bcb not found: ID does not exist" containerID="c28cfbfabc772b3ed0af12640598e35d422138fe420beef8bacf1a16879d1bcb"
Oct 30 22:30:31.683263 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:31.683177    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c28cfbfabc772b3ed0af12640598e35d422138fe420beef8bacf1a16879d1bcb"} err="failed to get container status \"c28cfbfabc772b3ed0af12640598e35d422138fe420beef8bacf1a16879d1bcb\": rpc error: code = NotFound desc = could not find container \"c28cfbfabc772b3ed0af12640598e35d422138fe420beef8bacf1a16879d1bcb\": container with ID starting with c28cfbfabc772b3ed0af12640598e35d422138fe420beef8bacf1a16879d1bcb not found: ID does not exist"
Oct 30 22:30:31.683263 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:31.683199    5061 scope.go:122] "RemoveContainer" containerID="37ba9b5cd8f7b071d28c86233af2bdb00bb896124c04340e88670ad1642596b3"
Oct 30 22:30:31.692408 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:31.692391    5061 scope.go:122] "RemoveContainer" containerID="37ba9b5cd8f7b071d28c86233af2bdb00bb896124c04340e88670ad1642596b3"
Oct 30 22:30:31.692662 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:30:31.692644    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37ba9b5cd8f7b071d28c86233af2bdb00bb896124c04340e88670ad1642596b3\": container with ID starting with 37ba9b5cd8f7b071d28c86233af2bdb00bb896124c04340e88670ad1642596b3 not found: ID does not exist" containerID="37ba9b5cd8f7b071d28c86233af2bdb00bb896124c04340e88670ad1642596b3"
Oct 30 22:30:31.692726 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:31.692671    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37ba9b5cd8f7b071d28c86233af2bdb00bb896124c04340e88670ad1642596b3"} err="failed to get container status \"37ba9b5cd8f7b071d28c86233af2bdb00bb896124c04340e88670ad1642596b3\": rpc error: code = NotFound desc = could not find container \"37ba9b5cd8f7b071d28c86233af2bdb00bb896124c04340e88670ad1642596b3\": container with ID starting with 37ba9b5cd8f7b071d28c86233af2bdb00bb896124c04340e88670ad1642596b3 not found: ID does not exist"
Oct 30 22:30:31.780536 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:31.780468    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-4d8041-r6xgc" podStartSLOduration=1.780449619 podStartE2EDuration="1.780449619s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:31.779378835 +0000 UTC m=+2885.392148595" watchObservedRunningTime="2026-10-30 22:30:31.780449619 +0000 UTC m=+2885.393219377"
Oct 30 22:30:31.886000 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:31.885960    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-84a3cf-6cb65fb6cf-r64nq"]
Oct 30 22:30:31.890523 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:31.890496    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-84a3cf-6cb65fb6cf-r64nq"]
Oct 30 22:30:31.993841 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:31.993768    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-84a3cf-6c95488bd4-cjmzh"]
Oct 30 22:30:31.998342 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:31.998309    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-84a3cf-6c95488bd4-cjmzh"]
Oct 30 22:30:32.534951 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:32.534910    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63cdc87a-9959-41f0-b5c8-1df23cf0cda4" path="/var/lib/kubelet/pods/63cdc87a-9959-41f0-b5c8-1df23cf0cda4/volumes"
Oct 30 22:30:32.535593 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:32.535580    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e098fc6-5863-4aa7-a134-f70fc5380a0d" path="/var/lib/kubelet/pods/7e098fc6-5863-4aa7-a134-f70fc5380a0d/volumes"
Oct 30 22:30:32.684801 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:32.684769    5061 generic.go:356] "Generic (PLEG): container finished" podID="1711e053-6008-4125-8479-502130dcecae" containerID="2affb77087b92c8c42090ae65d4419ef2ab4e1c187fcd8c22244217cb1bf9fd5" exitCode=0
Oct 30 22:30:32.685173 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:32.684828    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-4d8041-r6xgc" event={"ID":"1711e053-6008-4125-8479-502130dcecae","Type":"ContainerDied","Data":"2affb77087b92c8c42090ae65d4419ef2ab4e1c187fcd8c22244217cb1bf9fd5"}
Oct 30 22:30:33.829290 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:33.829265    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-4d8041-r6xgc"
Oct 30 22:30:33.979427 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:33.979341    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1711e053-6008-4125-8479-502130dcecae-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1711e053-6008-4125-8479-502130dcecae-maas-api-xks-ca\") pod \"1711e053-6008-4125-8479-502130dcecae\" (UID: \"1711e053-6008-4125-8479-502130dcecae\") "
Oct 30 22:30:33.979427 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:33.979390    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1711e053-6008-4125-8479-502130dcecae-kube-api-access-24mw9\" (UniqueName: \"kubernetes.io/projected/1711e053-6008-4125-8479-502130dcecae-kube-api-access-24mw9\") pod \"1711e053-6008-4125-8479-502130dcecae\" (UID: \"1711e053-6008-4125-8479-502130dcecae\") "
Oct 30 22:30:33.979710 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:33.979588    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1711e053-6008-4125-8479-502130dcecae-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1711e053-6008-4125-8479-502130dcecae-maas-api-openshift-ca\") pod \"1711e053-6008-4125-8479-502130dcecae\" (UID: \"1711e053-6008-4125-8479-502130dcecae\") "
Oct 30 22:30:33.979942 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:33.979914    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1711e053-6008-4125-8479-502130dcecae-maas-api-openshift-ca" pod "1711e053-6008-4125-8479-502130dcecae" (UID: "1711e053-6008-4125-8479-502130dcecae"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:33.981542 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:33.981507    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1711e053-6008-4125-8479-502130dcecae-maas-api-xks-ca" pod "1711e053-6008-4125-8479-502130dcecae" (UID: "1711e053-6008-4125-8479-502130dcecae"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:30:33.981665 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:33.981572    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1711e053-6008-4125-8479-502130dcecae-kube-api-access-24mw9" pod "1711e053-6008-4125-8479-502130dcecae" (UID: "1711e053-6008-4125-8479-502130dcecae"). InnerVolumeSpecName "kube-api-access-24mw9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:34.080451 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:34.080411    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1711e053-6008-4125-8479-502130dcecae-maas-api-xks-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:34.080451 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:34.080443    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24mw9\" (UniqueName: \"kubernetes.io/projected/1711e053-6008-4125-8479-502130dcecae-kube-api-access-24mw9\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:34.080451 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:34.080457    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1711e053-6008-4125-8479-502130dcecae-maas-api-openshift-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:34.697551 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:34.697513    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-4d8041-r6xgc"
Oct 30 22:30:34.697755 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:34.697512    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-4d8041-r6xgc" event={"ID":"1711e053-6008-4125-8479-502130dcecae","Type":"ContainerDied","Data":"f81359317c1be0abb4591c6267e755adfa5b732696717ca88a8391a96fb8953b"}
Oct 30 22:30:34.697755 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:34.697653    5061 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f81359317c1be0abb4591c6267e755adfa5b732696717ca88a8391a96fb8953b"
Oct 30 22:30:35.265908 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:35.265872    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-zhvck"]
Oct 30 22:30:35.266397 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:35.266111    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-zhvck" podUID="bbba8625-2b59-4055-a0f2-5746421672e2" containerName="istio-proxy" containerID="cri-o://a356d357861c378882837d0e3ea0c0599ff147cd3e7a7425a9ca5bf6e6e7a170" gracePeriod=30
Oct 30 22:30:40.541477 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:40.541422    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-zhvck"
Oct 30 22:30:40.646161 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:40.646122    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbba8625-2b59-4055-a0f2-5746421672e2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbba8625-2b59-4055-a0f2-5746421672e2-workload-certs\") pod \"bbba8625-2b59-4055-a0f2-5746421672e2\" (UID: \"bbba8625-2b59-4055-a0f2-5746421672e2\") "
Oct 30 22:30:40.646342 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:40.646206    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbba8625-2b59-4055-a0f2-5746421672e2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbba8625-2b59-4055-a0f2-5746421672e2-workload-socket\") pod \"bbba8625-2b59-4055-a0f2-5746421672e2\" (UID: \"bbba8625-2b59-4055-a0f2-5746421672e2\") "
Oct 30 22:30:40.646342 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:40.646259    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbba8625-2b59-4055-a0f2-5746421672e2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbba8625-2b59-4055-a0f2-5746421672e2-istio-envoy\") pod \"bbba8625-2b59-4055-a0f2-5746421672e2\" (UID: \"bbba8625-2b59-4055-a0f2-5746421672e2\") "
Oct 30 22:30:40.646342 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:40.646291    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbba8625-2b59-4055-a0f2-5746421672e2-kube-api-access-h9gfb\" (UniqueName: \"kubernetes.io/projected/bbba8625-2b59-4055-a0f2-5746421672e2-kube-api-access-h9gfb\") pod \"bbba8625-2b59-4055-a0f2-5746421672e2\" (UID: \"bbba8625-2b59-4055-a0f2-5746421672e2\") "
Oct 30 22:30:40.646470 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:40.646384    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbba8625-2b59-4055-a0f2-5746421672e2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbba8625-2b59-4055-a0f2-5746421672e2-istio-data\") pod \"bbba8625-2b59-4055-a0f2-5746421672e2\" (UID: \"bbba8625-2b59-4055-a0f2-5746421672e2\") "
Oct 30 22:30:40.646470 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:40.646415    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbba8625-2b59-4055-a0f2-5746421672e2-istio-token\" (UniqueName: \"kubernetes.io/projected/bbba8625-2b59-4055-a0f2-5746421672e2-istio-token\") pod \"bbba8625-2b59-4055-a0f2-5746421672e2\" (UID: \"bbba8625-2b59-4055-a0f2-5746421672e2\") "
Oct 30 22:30:40.646596 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:40.646472    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbba8625-2b59-4055-a0f2-5746421672e2-workload-certs" pod "bbba8625-2b59-4055-a0f2-5746421672e2" (UID: "bbba8625-2b59-4055-a0f2-5746421672e2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:40.646596 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:40.646490    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bbba8625-2b59-4055-a0f2-5746421672e2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbba8625-2b59-4055-a0f2-5746421672e2-istio-podinfo\") pod \"bbba8625-2b59-4055-a0f2-5746421672e2\" (UID: \"bbba8625-2b59-4055-a0f2-5746421672e2\") "
Oct 30 22:30:40.646596 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:40.646522    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbba8625-2b59-4055-a0f2-5746421672e2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbba8625-2b59-4055-a0f2-5746421672e2-credential-socket\") pod \"bbba8625-2b59-4055-a0f2-5746421672e2\" (UID: \"bbba8625-2b59-4055-a0f2-5746421672e2\") "
Oct 30 22:30:40.646596 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:40.646584    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbba8625-2b59-4055-a0f2-5746421672e2-workload-socket" pod "bbba8625-2b59-4055-a0f2-5746421672e2" (UID: "bbba8625-2b59-4055-a0f2-5746421672e2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:40.646827 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:40.646604    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbba8625-2b59-4055-a0f2-5746421672e2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbba8625-2b59-4055-a0f2-5746421672e2-istiod-ca-cert\") pod \"bbba8625-2b59-4055-a0f2-5746421672e2\" (UID: \"bbba8625-2b59-4055-a0f2-5746421672e2\") "
Oct 30 22:30:40.646827 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:40.646690    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbba8625-2b59-4055-a0f2-5746421672e2-istio-data" pod "bbba8625-2b59-4055-a0f2-5746421672e2" (UID: "bbba8625-2b59-4055-a0f2-5746421672e2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:40.646949 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:40.646904    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbba8625-2b59-4055-a0f2-5746421672e2-credential-socket" pod "bbba8625-2b59-4055-a0f2-5746421672e2" (UID: "bbba8625-2b59-4055-a0f2-5746421672e2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:40.647007 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:40.646988    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbba8625-2b59-4055-a0f2-5746421672e2-istiod-ca-cert" pod "bbba8625-2b59-4055-a0f2-5746421672e2" (UID: "bbba8625-2b59-4055-a0f2-5746421672e2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:40.647241 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:40.647169    5061 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbba8625-2b59-4055-a0f2-5746421672e2-credential-socket\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:40.647241 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:40.647195    5061 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbba8625-2b59-4055-a0f2-5746421672e2-istiod-ca-cert\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:40.647241 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:40.647210    5061 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbba8625-2b59-4055-a0f2-5746421672e2-workload-certs\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:40.647496 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:40.647239    5061 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbba8625-2b59-4055-a0f2-5746421672e2-workload-socket\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:40.647496 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:40.647300    5061 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbba8625-2b59-4055-a0f2-5746421672e2-istio-data\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:40.648838 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:40.648806    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbba8625-2b59-4055-a0f2-5746421672e2-istio-envoy" pod "bbba8625-2b59-4055-a0f2-5746421672e2" (UID: "bbba8625-2b59-4055-a0f2-5746421672e2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:40.648999 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:40.648981    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbba8625-2b59-4055-a0f2-5746421672e2-kube-api-access-h9gfb" pod "bbba8625-2b59-4055-a0f2-5746421672e2" (UID: "bbba8625-2b59-4055-a0f2-5746421672e2"). InnerVolumeSpecName "kube-api-access-h9gfb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:40.649147 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:40.649125    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bbba8625-2b59-4055-a0f2-5746421672e2-istio-podinfo" pod "bbba8625-2b59-4055-a0f2-5746421672e2" (UID: "bbba8625-2b59-4055-a0f2-5746421672e2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:30:40.649147 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:40.649138    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbba8625-2b59-4055-a0f2-5746421672e2-istio-token" pod "bbba8625-2b59-4055-a0f2-5746421672e2" (UID: "bbba8625-2b59-4055-a0f2-5746421672e2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:40.732605 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:40.732564    5061 generic.go:356] "Generic (PLEG): container finished" podID="bbba8625-2b59-4055-a0f2-5746421672e2" containerID="a356d357861c378882837d0e3ea0c0599ff147cd3e7a7425a9ca5bf6e6e7a170" exitCode=0
Oct 30 22:30:40.732803 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:40.732617    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-zhvck"
Oct 30 22:30:40.732803 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:40.732654    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-zhvck" event={"ID":"bbba8625-2b59-4055-a0f2-5746421672e2","Type":"ContainerDied","Data":"a356d357861c378882837d0e3ea0c0599ff147cd3e7a7425a9ca5bf6e6e7a170"}
Oct 30 22:30:40.732803 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:40.732724    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-zhvck" event={"ID":"bbba8625-2b59-4055-a0f2-5746421672e2","Type":"ContainerDied","Data":"0e6d71eb1667ae849ce1cce091dcaa791a3fc21cdd570d664337141deaa78d6a"}
Oct 30 22:30:40.732803 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:40.732754    5061 scope.go:122] "RemoveContainer" containerID="a356d357861c378882837d0e3ea0c0599ff147cd3e7a7425a9ca5bf6e6e7a170"
Oct 30 22:30:40.747648 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:40.746879    5061 scope.go:122] "RemoveContainer" containerID="a356d357861c378882837d0e3ea0c0599ff147cd3e7a7425a9ca5bf6e6e7a170"
Oct 30 22:30:40.747890 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:30:40.747861    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a356d357861c378882837d0e3ea0c0599ff147cd3e7a7425a9ca5bf6e6e7a170\": container with ID starting with a356d357861c378882837d0e3ea0c0599ff147cd3e7a7425a9ca5bf6e6e7a170 not found: ID does not exist" containerID="a356d357861c378882837d0e3ea0c0599ff147cd3e7a7425a9ca5bf6e6e7a170"
Oct 30 22:30:40.747988 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:40.747901    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a356d357861c378882837d0e3ea0c0599ff147cd3e7a7425a9ca5bf6e6e7a170"} err="failed to get container status \"a356d357861c378882837d0e3ea0c0599ff147cd3e7a7425a9ca5bf6e6e7a170\": rpc error: code = NotFound desc = could not find container \"a356d357861c378882837d0e3ea0c0599ff147cd3e7a7425a9ca5bf6e6e7a170\": container with ID starting with a356d357861c378882837d0e3ea0c0599ff147cd3e7a7425a9ca5bf6e6e7a170 not found: ID does not exist"
Oct 30 22:30:40.748309 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:40.748288    5061 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbba8625-2b59-4055-a0f2-5746421672e2-istio-envoy\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:40.748309 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:40.748308    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9gfb\" (UniqueName: \"kubernetes.io/projected/bbba8625-2b59-4055-a0f2-5746421672e2-kube-api-access-h9gfb\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:40.748450 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:40.748318    5061 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbba8625-2b59-4055-a0f2-5746421672e2-istio-token\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:40.748450 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:40.748327    5061 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbba8625-2b59-4055-a0f2-5746421672e2-istio-podinfo\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:40.854016 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:40.853983    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-zhvck"]
Oct 30 22:30:40.857517 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:40.857477    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-zhvck"]
Oct 30 22:30:41.535578 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:41.535517    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-zhvck container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:30:41.535795 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:41.535613    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-zhvck" podUID="bbba8625-2b59-4055-a0f2-5746421672e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:30:42.527240 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:42.527207    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbba8625-2b59-4055-a0f2-5746421672e2" path="/var/lib/kubelet/pods/bbba8625-2b59-4055-a0f2-5746421672e2/volumes"
Oct 30 22:30:47.913125 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:47.913085    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-4d8041-r6xgc"]
Oct 30 22:30:47.960158 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:47.960125    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-4d8041-r6xgc"]
Oct 30 22:30:48.527965 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:48.527929    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1711e053-6008-4125-8479-502130dcecae" path="/var/lib/kubelet/pods/1711e053-6008-4125-8479-502130dcecae/volumes"
Oct 30 22:30:54.668495 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:30:54.668447    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 22:30:55.469610 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:55.469574    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-02bd3b-xp2zq"]
Oct 30 22:30:55.470014 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:55.470002    5061 cpu_manager.go:414] "containerMap: removing container" podUID="1711e053-6008-4125-8479-502130dcecae" containerName="revoke-keys"
Oct 30 22:30:55.470060 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:55.470016    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1711e053-6008-4125-8479-502130dcecae" containerName="revoke-keys"
Oct 30 22:30:55.470060 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:55.470029    5061 cpu_manager.go:414] "containerMap: removing container" podUID="333a34f7-9655-4ea6-9588-b9f5cb8fe305" containerName="cleanup"
Oct 30 22:30:55.470060 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:55.470034    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="333a34f7-9655-4ea6-9588-b9f5cb8fe305" containerName="cleanup"
Oct 30 22:30:55.470060 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:55.470040    5061 cpu_manager.go:414] "containerMap: removing container" podUID="bbba8625-2b59-4055-a0f2-5746421672e2" containerName="istio-proxy"
Oct 30 22:30:55.470060 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:55.470045    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbba8625-2b59-4055-a0f2-5746421672e2" containerName="istio-proxy"
Oct 30 22:30:55.470060 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:55.470054    5061 cpu_manager.go:414] "containerMap: removing container" podUID="7e098fc6-5863-4aa7-a134-f70fc5380a0d" containerName="payload-processing"
Oct 30 22:30:55.470060 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:55.470059    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e098fc6-5863-4aa7-a134-f70fc5380a0d" containerName="payload-processing"
Oct 30 22:30:55.470297 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:55.470065    5061 cpu_manager.go:414] "containerMap: removing container" podUID="63cdc87a-9959-41f0-b5c8-1df23cf0cda4" containerName="payload-pre-processing"
Oct 30 22:30:55.470297 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:55.470070    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63cdc87a-9959-41f0-b5c8-1df23cf0cda4" containerName="payload-pre-processing"
Oct 30 22:30:55.470297 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:55.470101    5061 cpu_manager.go:414] "containerMap: removing container" podUID="51015ddd-3122-4cfb-99bc-d4c041646c64" containerName="cleanup"
Oct 30 22:30:55.470297 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:55.470106    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51015ddd-3122-4cfb-99bc-d4c041646c64" containerName="cleanup"
Oct 30 22:30:55.470297 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:55.470114    5061 cpu_manager.go:414] "containerMap: removing container" podUID="d14224db-1f4f-45c3-a36b-8af75018a006" containerName="cleanup"
Oct 30 22:30:55.470297 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:55.470118    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d14224db-1f4f-45c3-a36b-8af75018a006" containerName="cleanup"
Oct 30 22:30:55.470297 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:55.470188    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="63cdc87a-9959-41f0-b5c8-1df23cf0cda4" containerName="payload-pre-processing"
Oct 30 22:30:55.470297 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:55.470204    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="1711e053-6008-4125-8479-502130dcecae" containerName="revoke-keys"
Oct 30 22:30:55.470297 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:55.470210    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cf03377-68fe-44d3-a000-3a4893ef6b8a" containerName="istio-proxy"
Oct 30 22:30:55.470297 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:55.470216    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbba8625-2b59-4055-a0f2-5746421672e2" containerName="istio-proxy"
Oct 30 22:30:55.470297 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:55.470221    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e098fc6-5863-4aa7-a134-f70fc5380a0d" containerName="payload-processing"
Oct 30 22:30:55.474949 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:55.474927    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-02bd3b-xp2zq"
Oct 30 22:30:55.477943 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:55.477917    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cbd0913-79ca-4da0-a84f-507bd8ce1010-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-02bd3b-xp2zq\" (UID: \"2cbd0913-79ca-4da0-a84f-507bd8ce1010\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-02bd3b-xp2zq"
Oct 30 22:30:55.478078 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:55.477951    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cbd0913-79ca-4da0-a84f-507bd8ce1010-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-02bd3b-xp2zq\" (UID: \"2cbd0913-79ca-4da0-a84f-507bd8ce1010\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-02bd3b-xp2zq"
Oct 30 22:30:55.478156 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:55.478074    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4l4n\" (UniqueName: \"kubernetes.io/projected/2cbd0913-79ca-4da0-a84f-507bd8ce1010-kube-api-access-d4l4n\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-02bd3b-xp2zq\" (UID: \"2cbd0913-79ca-4da0-a84f-507bd8ce1010\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-02bd3b-xp2zq"
Oct 30 22:30:55.524724 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:55.524699    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:30:55.571571 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:55.571539    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:30:55.571733 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:55.571600    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gdmfg\""
Oct 30 22:30:55.571733 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:55.571605    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:30:55.573899 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:55.573876    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-02bd3b-xp2zq"]
Oct 30 22:30:55.578974 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:55.578958    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cbd0913-79ca-4da0-a84f-507bd8ce1010-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-02bd3b-xp2zq\" (UID: \"2cbd0913-79ca-4da0-a84f-507bd8ce1010\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-02bd3b-xp2zq"
Oct 30 22:30:55.579043 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:55.578981    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cbd0913-79ca-4da0-a84f-507bd8ce1010-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-02bd3b-xp2zq\" (UID: \"2cbd0913-79ca-4da0-a84f-507bd8ce1010\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-02bd3b-xp2zq"
Oct 30 22:30:55.579043 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:55.579034    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4l4n\" (UniqueName: \"kubernetes.io/projected/2cbd0913-79ca-4da0-a84f-507bd8ce1010-kube-api-access-d4l4n\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-02bd3b-xp2zq\" (UID: \"2cbd0913-79ca-4da0-a84f-507bd8ce1010\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-02bd3b-xp2zq"
Oct 30 22:30:55.579690 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:55.579665    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cbd0913-79ca-4da0-a84f-507bd8ce1010-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-02bd3b-xp2zq\" (UID: \"2cbd0913-79ca-4da0-a84f-507bd8ce1010\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-02bd3b-xp2zq"
Oct 30 22:30:55.581292 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:55.581275    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cbd0913-79ca-4da0-a84f-507bd8ce1010-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-02bd3b-xp2zq\" (UID: \"2cbd0913-79ca-4da0-a84f-507bd8ce1010\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-02bd3b-xp2zq"
Oct 30 22:30:55.633766 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:55.633715    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4l4n\" (UniqueName: \"kubernetes.io/projected/2cbd0913-79ca-4da0-a84f-507bd8ce1010-kube-api-access-d4l4n\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-02bd3b-xp2zq\" (UID: \"2cbd0913-79ca-4da0-a84f-507bd8ce1010\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-02bd3b-xp2zq"
Oct 30 22:30:55.788182 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:55.788096    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-02bd3b-xp2zq"
Oct 30 22:30:56.153500 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:56.153467    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-02bd3b-xp2zq"]
Oct 30 22:30:56.155336 ip-10-0-130-60 kubenswrapper[5061]: W1030 22:30:56.155307    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cbd0913_79ca_4da0_a84f_507bd8ce1010.slice/crio-f4a5cbf62725c0127a1f334b06fcc6e744bd2d8aa3fba234095ad402fb261cd5 WatchSource:0}: Error finding container f4a5cbf62725c0127a1f334b06fcc6e744bd2d8aa3fba234095ad402fb261cd5: Status 404 returned error can't find the container with id f4a5cbf62725c0127a1f334b06fcc6e744bd2d8aa3fba234095ad402fb261cd5
Oct 30 22:30:56.821470 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:56.821436    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-02bd3b-xp2zq" event={"ID":"2cbd0913-79ca-4da0-a84f-507bd8ce1010","Type":"ContainerStarted","Data":"ef8f004856ad46c8dd2bb240b23a4f4717154145c492819c66d8ebf245641cc7"}
Oct 30 22:30:56.821873 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:56.821489    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-02bd3b-xp2zq" event={"ID":"2cbd0913-79ca-4da0-a84f-507bd8ce1010","Type":"ContainerStarted","Data":"f4a5cbf62725c0127a1f334b06fcc6e744bd2d8aa3fba234095ad402fb261cd5"}
Oct 30 22:30:56.929707 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:56.929655    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-02bd3b-xp2zq" podStartSLOduration=1.9296427870000001 podStartE2EDuration="1.929642787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:56.928655776 +0000 UTC m=+2910.541425534" watchObservedRunningTime="2026-10-30 22:30:56.929642787 +0000 UTC m=+2910.542412585"
Oct 30 22:30:57.831988 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:57.831946    5061 generic.go:356] "Generic (PLEG): container finished" podID="2cbd0913-79ca-4da0-a84f-507bd8ce1010" containerID="ef8f004856ad46c8dd2bb240b23a4f4717154145c492819c66d8ebf245641cc7" exitCode=0
Oct 30 22:30:57.832455 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:57.831997    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-02bd3b-xp2zq" event={"ID":"2cbd0913-79ca-4da0-a84f-507bd8ce1010","Type":"ContainerDied","Data":"ef8f004856ad46c8dd2bb240b23a4f4717154145c492819c66d8ebf245641cc7"}
Oct 30 22:30:58.983612 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:58.983585    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-02bd3b-xp2zq"
Oct 30 22:30:59.014797 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:59.014771    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2cbd0913-79ca-4da0-a84f-507bd8ce1010-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cbd0913-79ca-4da0-a84f-507bd8ce1010-maas-api-xks-ca\") pod \"2cbd0913-79ca-4da0-a84f-507bd8ce1010\" (UID: \"2cbd0913-79ca-4da0-a84f-507bd8ce1010\") "
Oct 30 22:30:59.014914 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:59.014807    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2cbd0913-79ca-4da0-a84f-507bd8ce1010-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cbd0913-79ca-4da0-a84f-507bd8ce1010-maas-api-openshift-ca\") pod \"2cbd0913-79ca-4da0-a84f-507bd8ce1010\" (UID: \"2cbd0913-79ca-4da0-a84f-507bd8ce1010\") "
Oct 30 22:30:59.014914 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:59.014828    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cbd0913-79ca-4da0-a84f-507bd8ce1010-kube-api-access-d4l4n\" (UniqueName: \"kubernetes.io/projected/2cbd0913-79ca-4da0-a84f-507bd8ce1010-kube-api-access-d4l4n\") pod \"2cbd0913-79ca-4da0-a84f-507bd8ce1010\" (UID: \"2cbd0913-79ca-4da0-a84f-507bd8ce1010\") "
Oct 30 22:30:59.015193 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:59.015167    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cbd0913-79ca-4da0-a84f-507bd8ce1010-maas-api-openshift-ca" pod "2cbd0913-79ca-4da0-a84f-507bd8ce1010" (UID: "2cbd0913-79ca-4da0-a84f-507bd8ce1010"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:59.016787 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:59.016768    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2cbd0913-79ca-4da0-a84f-507bd8ce1010-maas-api-xks-ca" pod "2cbd0913-79ca-4da0-a84f-507bd8ce1010" (UID: "2cbd0913-79ca-4da0-a84f-507bd8ce1010"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:30:59.016890 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:59.016866    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cbd0913-79ca-4da0-a84f-507bd8ce1010-kube-api-access-d4l4n" pod "2cbd0913-79ca-4da0-a84f-507bd8ce1010" (UID: "2cbd0913-79ca-4da0-a84f-507bd8ce1010"). InnerVolumeSpecName "kube-api-access-d4l4n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:59.116089 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:59.116013    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cbd0913-79ca-4da0-a84f-507bd8ce1010-maas-api-xks-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:59.116089 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:59.116054    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cbd0913-79ca-4da0-a84f-507bd8ce1010-maas-api-openshift-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:59.116089 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:59.116071    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4l4n\" (UniqueName: \"kubernetes.io/projected/2cbd0913-79ca-4da0-a84f-507bd8ce1010-kube-api-access-d4l4n\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:59.844984 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:59.844948    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-02bd3b-xp2zq" event={"ID":"2cbd0913-79ca-4da0-a84f-507bd8ce1010","Type":"ContainerDied","Data":"f4a5cbf62725c0127a1f334b06fcc6e744bd2d8aa3fba234095ad402fb261cd5"}
Oct 30 22:30:59.845196 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:59.845010    5061 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f4a5cbf62725c0127a1f334b06fcc6e744bd2d8aa3fba234095ad402fb261cd5"
Oct 30 22:30:59.845196 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:30:59.844961    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-02bd3b-xp2zq"
Oct 30 22:31:18.425113 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:31:18.425009    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-02bd3b-xp2zq"]
Oct 30 22:31:18.471033 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:31:18.470990    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-02bd3b-xp2zq"]
Oct 30 22:31:18.533136 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:31:18.533097    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2cbd0913-79ca-4da0-a84f-507bd8ce1010" path="/var/lib/kubelet/pods/2cbd0913-79ca-4da0-a84f-507bd8ce1010/volumes"
Oct 30 22:31:24.666607 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:31:24.666566    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 22:31:25.786294 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:31:25.786258    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-5402dd-727h9"]
Oct 30 22:31:25.786755 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:31:25.786740    5061 cpu_manager.go:414] "containerMap: removing container" podUID="2cbd0913-79ca-4da0-a84f-507bd8ce1010" containerName="revoke-keys"
Oct 30 22:31:25.786807 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:31:25.786757    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cbd0913-79ca-4da0-a84f-507bd8ce1010" containerName="revoke-keys"
Oct 30 22:31:25.786848 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:31:25.786837    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cbd0913-79ca-4da0-a84f-507bd8ce1010" containerName="revoke-keys"
Oct 30 22:31:25.791456 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:31:25.791438    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-5402dd-727h9"
Oct 30 22:31:25.841096 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:31:25.841066    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:31:25.871658 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:31:25.871619    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f0a0bbe-353b-4adc-ba93-49e980fd3608-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-5402dd-727h9\" (UID: \"8f0a0bbe-353b-4adc-ba93-49e980fd3608\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-5402dd-727h9"
Oct 30 22:31:25.871839 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:31:25.871810    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4dtq\" (UniqueName: \"kubernetes.io/projected/8f0a0bbe-353b-4adc-ba93-49e980fd3608-kube-api-access-x4dtq\") pod \"maas-api-revoke-keys-e2e-worker-w5-5402dd-727h9\" (UID: \"8f0a0bbe-353b-4adc-ba93-49e980fd3608\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-5402dd-727h9"
Oct 30 22:31:25.871941 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:31:25.871921    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f0a0bbe-353b-4adc-ba93-49e980fd3608-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-5402dd-727h9\" (UID: \"8f0a0bbe-353b-4adc-ba93-49e980fd3608\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-5402dd-727h9"
Oct 30 22:31:25.888195 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:31:25.888173    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:31:25.888195 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:31:25.888178    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:31:25.888355 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:31:25.888185    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gdmfg\""
Oct 30 22:31:25.890461 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:31:25.890442    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-5402dd-727h9"]
Oct 30 22:31:25.973428 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:31:25.973392    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f0a0bbe-353b-4adc-ba93-49e980fd3608-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-5402dd-727h9\" (UID: \"8f0a0bbe-353b-4adc-ba93-49e980fd3608\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-5402dd-727h9"
Oct 30 22:31:25.973656 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:31:25.973481    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4dtq\" (UniqueName: \"kubernetes.io/projected/8f0a0bbe-353b-4adc-ba93-49e980fd3608-kube-api-access-x4dtq\") pod \"maas-api-revoke-keys-e2e-worker-w5-5402dd-727h9\" (UID: \"8f0a0bbe-353b-4adc-ba93-49e980fd3608\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-5402dd-727h9"
Oct 30 22:31:25.973656 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:31:25.973517    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f0a0bbe-353b-4adc-ba93-49e980fd3608-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-5402dd-727h9\" (UID: \"8f0a0bbe-353b-4adc-ba93-49e980fd3608\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-5402dd-727h9"
Oct 30 22:31:25.974153 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:31:25.974126    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f0a0bbe-353b-4adc-ba93-49e980fd3608-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-5402dd-727h9\" (UID: \"8f0a0bbe-353b-4adc-ba93-49e980fd3608\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-5402dd-727h9"
Oct 30 22:31:25.975867 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:31:25.975842    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f0a0bbe-353b-4adc-ba93-49e980fd3608-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-5402dd-727h9\" (UID: \"8f0a0bbe-353b-4adc-ba93-49e980fd3608\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-5402dd-727h9"
Oct 30 22:31:26.027865 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:31:26.027836    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4dtq\" (UniqueName: \"kubernetes.io/projected/8f0a0bbe-353b-4adc-ba93-49e980fd3608-kube-api-access-x4dtq\") pod \"maas-api-revoke-keys-e2e-worker-w5-5402dd-727h9\" (UID: \"8f0a0bbe-353b-4adc-ba93-49e980fd3608\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-5402dd-727h9"
Oct 30 22:31:26.104075 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:31:26.104047    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-5402dd-727h9"
Oct 30 22:31:26.467768 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:31:26.467737    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-5402dd-727h9"]
Oct 30 22:31:26.469071 ip-10-0-130-60 kubenswrapper[5061]: W1030 22:31:26.469036    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f0a0bbe_353b_4adc_ba93_49e980fd3608.slice/crio-0d2b065eda0f27fe01a3396c51e2b28ab7adb113e81f789138d5a9bf305b0605 WatchSource:0}: Error finding container 0d2b065eda0f27fe01a3396c51e2b28ab7adb113e81f789138d5a9bf305b0605: Status 404 returned error can't find the container with id 0d2b065eda0f27fe01a3396c51e2b28ab7adb113e81f789138d5a9bf305b0605
Oct 30 22:31:26.998041 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:31:26.997941    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-5402dd-727h9" event={"ID":"8f0a0bbe-353b-4adc-ba93-49e980fd3608","Type":"ContainerStarted","Data":"0ea7e6cd635be4b5d9960f6989a6f918b5fa5dafad74f9b99255efaf414aa511"}
Oct 30 22:31:26.998041 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:31:26.997999    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-5402dd-727h9" event={"ID":"8f0a0bbe-353b-4adc-ba93-49e980fd3608","Type":"ContainerStarted","Data":"0d2b065eda0f27fe01a3396c51e2b28ab7adb113e81f789138d5a9bf305b0605"}
Oct 30 22:31:27.107735 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:31:27.107676    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-5402dd-727h9" podStartSLOduration=2.107658722 podStartE2EDuration="2.107658722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:31:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:31:27.106674131 +0000 UTC m=+2940.719443907" watchObservedRunningTime="2026-10-30 22:31:27.107658722 +0000 UTC m=+2940.720428481"
Oct 30 22:31:28.005363 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:31:28.005323    5061 generic.go:356] "Generic (PLEG): container finished" podID="8f0a0bbe-353b-4adc-ba93-49e980fd3608" containerID="0ea7e6cd635be4b5d9960f6989a6f918b5fa5dafad74f9b99255efaf414aa511" exitCode=0
Oct 30 22:31:28.005786 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:31:28.005389    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-5402dd-727h9" event={"ID":"8f0a0bbe-353b-4adc-ba93-49e980fd3608","Type":"ContainerDied","Data":"0ea7e6cd635be4b5d9960f6989a6f918b5fa5dafad74f9b99255efaf414aa511"}
Oct 30 22:31:29.150843 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:31:29.150817    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-5402dd-727h9"
Oct 30 22:31:29.302981 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:31:29.302952    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f0a0bbe-353b-4adc-ba93-49e980fd3608-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f0a0bbe-353b-4adc-ba93-49e980fd3608-maas-api-xks-ca\") pod \"8f0a0bbe-353b-4adc-ba93-49e980fd3608\" (UID: \"8f0a0bbe-353b-4adc-ba93-49e980fd3608\") "
Oct 30 22:31:29.303154 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:31:29.303045    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f0a0bbe-353b-4adc-ba93-49e980fd3608-kube-api-access-x4dtq\" (UniqueName: \"kubernetes.io/projected/8f0a0bbe-353b-4adc-ba93-49e980fd3608-kube-api-access-x4dtq\") pod \"8f0a0bbe-353b-4adc-ba93-49e980fd3608\" (UID: \"8f0a0bbe-353b-4adc-ba93-49e980fd3608\") "
Oct 30 22:31:29.303154 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:31:29.303133    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f0a0bbe-353b-4adc-ba93-49e980fd3608-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f0a0bbe-353b-4adc-ba93-49e980fd3608-maas-api-openshift-ca\") pod \"8f0a0bbe-353b-4adc-ba93-49e980fd3608\" (UID: \"8f0a0bbe-353b-4adc-ba93-49e980fd3608\") "
Oct 30 22:31:29.303472 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:31:29.303446    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f0a0bbe-353b-4adc-ba93-49e980fd3608-maas-api-openshift-ca" pod "8f0a0bbe-353b-4adc-ba93-49e980fd3608" (UID: "8f0a0bbe-353b-4adc-ba93-49e980fd3608"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:31:29.303612 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:31:29.303572    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f0a0bbe-353b-4adc-ba93-49e980fd3608-maas-api-openshift-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:29.305146 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:31:29.305123    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f0a0bbe-353b-4adc-ba93-49e980fd3608-maas-api-xks-ca" pod "8f0a0bbe-353b-4adc-ba93-49e980fd3608" (UID: "8f0a0bbe-353b-4adc-ba93-49e980fd3608"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:31:29.305308 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:31:29.305162    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f0a0bbe-353b-4adc-ba93-49e980fd3608-kube-api-access-x4dtq" pod "8f0a0bbe-353b-4adc-ba93-49e980fd3608" (UID: "8f0a0bbe-353b-4adc-ba93-49e980fd3608"). InnerVolumeSpecName "kube-api-access-x4dtq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:31:29.393982 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:31:29.393954    5061 scope.go:122] "RemoveContainer" containerID="7b1113453a932d05bbd87be2a92526a1a2105119e09762f7c5c92398c2b4918a"
Oct 30 22:31:29.402965 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:31:29.402943    5061 scope.go:122] "RemoveContainer" containerID="60800004c41122a730f6d9afab0b76e0bb7318d24aa7b8f0f61646154eb89425"
Oct 30 22:31:29.404049 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:31:29.404025    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f0a0bbe-353b-4adc-ba93-49e980fd3608-maas-api-xks-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:29.404049 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:31:29.404048    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4dtq\" (UniqueName: \"kubernetes.io/projected/8f0a0bbe-353b-4adc-ba93-49e980fd3608-kube-api-access-x4dtq\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:29.410868 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:31:29.410843    5061 scope.go:122] "RemoveContainer" containerID="616fdec68611feded71059e1604adb7d684a5aec6071420cfeeaa26ee5d14046"
Oct 30 22:31:30.016995 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:31:30.016963    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-5402dd-727h9"
Oct 30 22:31:30.017210 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:31:30.016999    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-5402dd-727h9" event={"ID":"8f0a0bbe-353b-4adc-ba93-49e980fd3608","Type":"ContainerDied","Data":"0d2b065eda0f27fe01a3396c51e2b28ab7adb113e81f789138d5a9bf305b0605"}
Oct 30 22:31:30.017210 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:31:30.017049    5061 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0d2b065eda0f27fe01a3396c51e2b28ab7adb113e81f789138d5a9bf305b0605"
Oct 30 22:31:37.326085 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:31:37.326049    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-5402dd-57dc966bb-4x6rt"]
Oct 30 22:31:37.432813 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:31:37.432783    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-5402dd-86b6d5dd95-xzj89"]
Oct 30 22:31:42.282391 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:31:42.282357    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-718807d0-hl455"]
Oct 30 22:31:42.282874 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:31:42.282859    5061 cpu_manager.go:414] "containerMap: removing container" podUID="8f0a0bbe-353b-4adc-ba93-49e980fd3608" containerName="revoke-keys"
Oct 30 22:31:42.282927 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:31:42.282875    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f0a0bbe-353b-4adc-ba93-49e980fd3608" containerName="revoke-keys"
Oct 30 22:31:42.282964 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:31:42.282959    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f0a0bbe-353b-4adc-ba93-49e980fd3608" containerName="revoke-keys"
Oct 30 22:31:42.287779 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:31:42.287760    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-718807d0-hl455"
Oct 30 22:31:42.323852 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:31:42.323823    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2caef524-f1b2-4356-a253-66a13d80d8ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-718807d0-hl455\" (UID: \"2caef524-f1b2-4356-a253-66a13d80d8ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-718807d0-hl455"
Oct 30 22:31:42.323978 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:31:42.323954    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmr5q\" (UniqueName: \"kubernetes.io/projected/2caef524-f1b2-4356-a253-66a13d80d8ca-kube-api-access-qmr5q\") pod \"maas-api-revoke-sub-718807d0-hl455\" (UID: \"2caef524-f1b2-4356-a253-66a13d80d8ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-718807d0-hl455"
Oct 30 22:31:42.324049 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:31:42.323982    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2caef524-f1b2-4356-a253-66a13d80d8ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-718807d0-hl455\" (UID: \"2caef524-f1b2-4356-a253-66a13d80d8ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-718807d0-hl455"
Oct 30 22:31:42.326606 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:31:42.326581    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w5-5402dd-57dc966bb-4x6rt" podUID="a02d777c-062a-4182-81e7-1952fa5924f5" containerName="payload-processing" containerID="cri-o://5c6bc908e527fe0e7d23233fa6a218eeddb3b58cc9984d5095231dcdee2dc100" gracePeriod=25
Oct 30 22:31:42.346213 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:31:42.346185    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-718807d0-hl455"]
Oct 30 22:31:42.387755 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:31:42.387724    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:31:42.387898 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:31:42.387813    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:31:42.387898 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:31:42.387831    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gdmfg\""
Oct 30 22:31:42.388015 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:31:42.387914    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:31:42.424743 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:31:42.424707    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmr5q\" (UniqueName: \"kubernetes.io/projected/2caef524-f1b2-4356-a253-66a13d80d8ca-kube-api-access-qmr5q\") pod \"maas-api-revoke-sub-718807d0-hl455\" (UID: \"2caef524-f1b2-4356-a253-66a13d80d8ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-718807d0-hl455"
Oct 30 22:31:42.424743 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:31:42.424744    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2caef524-f1b2-4356-a253-66a13d80d8ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-718807d0-hl455\" (UID: \"2caef524-f1b2-4356-a253-66a13d80d8ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-718807d0-hl455"
Oct 30 22:31:42.424981 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:31:42.424789    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2caef524-f1b2-4356-a253-66a13d80d8ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-718807d0-hl455\" (UID: \"2caef524-f1b2-4356-a253-66a13d80d8ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-718807d0-hl455"
Oct 30 22:31:42.425499 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:31:42.425481    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2caef524-f1b2-4356-a253-66a13d80d8ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-718807d0-hl455\" (UID: \"2caef524-f1b2-4356-a253-66a13d80d8ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-718807d0-hl455"
Oct 30 22:31:42.427269 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:31:42.427249    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2caef524-f1b2-4356-a253-66a13d80d8ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-718807d0-hl455\" (UID: \"2caef524-f1b2-4356-a253-66a13d80d8ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-718807d0-hl455"
Oct 30 22:31:42.433131 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:31:42.433083    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-5402dd-86b6d5dd95-xzj89" podUID="61e905c3-97ef-4f17-aba7-0ea699720853" containerName="payload-pre-processing" containerID="cri-o://e8e3bf7f3a42e9c86d0d52fafd396750222aa6219defa39e2431640714f71120" gracePeriod=25
Oct 30 22:31:42.480305 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:31:42.480278    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmr5q\" (UniqueName: \"kubernetes.io/projected/2caef524-f1b2-4356-a253-66a13d80d8ca-kube-api-access-qmr5q\") pod \"maas-api-revoke-sub-718807d0-hl455\" (UID: \"2caef524-f1b2-4356-a253-66a13d80d8ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-718807d0-hl455"
Oct 30 22:31:42.600614 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:31:42.600587    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-718807d0-hl455"
Oct 30 22:31:42.968634 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:31:42.968610    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-718807d0-hl455"]
Oct 30 22:31:42.971089 ip-10-0-130-60 kubenswrapper[5061]: W1030 22:31:42.971061    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2caef524_f1b2_4356_a253_66a13d80d8ca.slice/crio-ae55170db767e84676e0965b9e0df6762712729c55c6b8a4013da168079aa257 WatchSource:0}: Error finding container ae55170db767e84676e0965b9e0df6762712729c55c6b8a4013da168079aa257: Status 404 returned error can't find the container with id ae55170db767e84676e0965b9e0df6762712729c55c6b8a4013da168079aa257
Oct 30 22:31:43.089585 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:31:43.089553    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-718807d0-hl455" event={"ID":"2caef524-f1b2-4356-a253-66a13d80d8ca","Type":"ContainerStarted","Data":"ae55170db767e84676e0965b9e0df6762712729c55c6b8a4013da168079aa257"}
Oct 30 22:31:44.097243 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:31:44.097210    5061 generic.go:356] "Generic (PLEG): container finished" podID="2caef524-f1b2-4356-a253-66a13d80d8ca" containerID="890a6158fc31b9e035338bcb1db222e09ce77c077ea8fea10b695ab0894ddb4b" exitCode=0
Oct 30 22:31:44.097654 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:31:44.097267    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-718807d0-hl455" event={"ID":"2caef524-f1b2-4356-a253-66a13d80d8ca","Type":"ContainerDied","Data":"890a6158fc31b9e035338bcb1db222e09ce77c077ea8fea10b695ab0894ddb4b"}
Oct 30 22:31:45.246509 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:31:45.246481    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-718807d0-hl455"
Oct 30 22:31:45.353687 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:31:45.353625    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2caef524-f1b2-4356-a253-66a13d80d8ca-kube-api-access-qmr5q\" (UniqueName: \"kubernetes.io/projected/2caef524-f1b2-4356-a253-66a13d80d8ca-kube-api-access-qmr5q\") pod \"2caef524-f1b2-4356-a253-66a13d80d8ca\" (UID: \"2caef524-f1b2-4356-a253-66a13d80d8ca\") "
Oct 30 22:31:45.353687 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:31:45.353677    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2caef524-f1b2-4356-a253-66a13d80d8ca-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2caef524-f1b2-4356-a253-66a13d80d8ca-maas-api-xks-ca\") pod \"2caef524-f1b2-4356-a253-66a13d80d8ca\" (UID: \"2caef524-f1b2-4356-a253-66a13d80d8ca\") "
Oct 30 22:31:45.353912 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:31:45.353718    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2caef524-f1b2-4356-a253-66a13d80d8ca-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2caef524-f1b2-4356-a253-66a13d80d8ca-maas-api-openshift-ca\") pod \"2caef524-f1b2-4356-a253-66a13d80d8ca\" (UID: \"2caef524-f1b2-4356-a253-66a13d80d8ca\") "
Oct 30 22:31:45.354119 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:31:45.354095    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2caef524-f1b2-4356-a253-66a13d80d8ca-maas-api-openshift-ca" pod "2caef524-f1b2-4356-a253-66a13d80d8ca" (UID: "2caef524-f1b2-4356-a253-66a13d80d8ca"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:31:45.355828 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:31:45.355809    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2caef524-f1b2-4356-a253-66a13d80d8ca-kube-api-access-qmr5q" pod "2caef524-f1b2-4356-a253-66a13d80d8ca" (UID: "2caef524-f1b2-4356-a253-66a13d80d8ca"). InnerVolumeSpecName "kube-api-access-qmr5q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:31:45.355949 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:31:45.355849    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2caef524-f1b2-4356-a253-66a13d80d8ca-maas-api-xks-ca" pod "2caef524-f1b2-4356-a253-66a13d80d8ca" (UID: "2caef524-f1b2-4356-a253-66a13d80d8ca"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:31:45.455059 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:31:45.455027    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmr5q\" (UniqueName: \"kubernetes.io/projected/2caef524-f1b2-4356-a253-66a13d80d8ca-kube-api-access-qmr5q\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:45.455059 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:31:45.455054    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2caef524-f1b2-4356-a253-66a13d80d8ca-maas-api-xks-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:45.455229 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:31:45.455065    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2caef524-f1b2-4356-a253-66a13d80d8ca-maas-api-openshift-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:45.741495 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:31:45.741408    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-5402dd-86b6d5dd95-xzj89" podUID="61e905c3-97ef-4f17-aba7-0ea699720853" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:31:45.741680 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:31:45.741408    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-5402dd-57dc966bb-4x6rt" podUID="a02d777c-062a-4182-81e7-1952fa5924f5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:31:46.114850 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:31:46.114817    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-718807d0-hl455"
Oct 30 22:31:46.115038 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:31:46.114849    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-718807d0-hl455" event={"ID":"2caef524-f1b2-4356-a253-66a13d80d8ca","Type":"ContainerDied","Data":"ae55170db767e84676e0965b9e0df6762712729c55c6b8a4013da168079aa257"}
Oct 30 22:31:46.115038 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:31:46.114941    5061 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ae55170db767e84676e0965b9e0df6762712729c55c6b8a4013da168079aa257"
Oct 30 22:31:54.666088 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:31:54.666055    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 22:31:55.741675 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:31:55.741636    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-5402dd-57dc966bb-4x6rt" podUID="a02d777c-062a-4182-81e7-1952fa5924f5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:31:55.742061 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:31:55.741703    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-5402dd-86b6d5dd95-xzj89" podUID="61e905c3-97ef-4f17-aba7-0ea699720853" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:31:59.732296 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:31:59.732250    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-5402dd-727h9"]
Oct 30 22:31:59.779390 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:31:59.779359    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-5402dd-727h9"]
Oct 30 22:32:00.527075 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:00.527036    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f0a0bbe-353b-4adc-ba93-49e980fd3608" path="/var/lib/kubelet/pods/8f0a0bbe-353b-4adc-ba93-49e980fd3608/volumes"
Oct 30 22:32:02.581221 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:02.581190    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-5402dd-57dc966bb-4x6rt"
Oct 30 22:32:02.625127 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:02.625092    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a02d777c-062a-4182-81e7-1952fa5924f5-kube-api-access-l7z59\" (UniqueName: \"kubernetes.io/projected/a02d777c-062a-4182-81e7-1952fa5924f5-kube-api-access-l7z59\") pod \"a02d777c-062a-4182-81e7-1952fa5924f5\" (UID: \"a02d777c-062a-4182-81e7-1952fa5924f5\") "
Oct 30 22:32:02.625309 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:02.625241    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a02d777c-062a-4182-81e7-1952fa5924f5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a02d777c-062a-4182-81e7-1952fa5924f5-plugins-config-volume\") pod \"a02d777c-062a-4182-81e7-1952fa5924f5\" (UID: \"a02d777c-062a-4182-81e7-1952fa5924f5\") "
Oct 30 22:32:02.625913 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:02.625871    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a02d777c-062a-4182-81e7-1952fa5924f5-plugins-config-volume" pod "a02d777c-062a-4182-81e7-1952fa5924f5" (UID: "a02d777c-062a-4182-81e7-1952fa5924f5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:32:02.627782 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:02.627753    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a02d777c-062a-4182-81e7-1952fa5924f5-kube-api-access-l7z59" pod "a02d777c-062a-4182-81e7-1952fa5924f5" (UID: "a02d777c-062a-4182-81e7-1952fa5924f5"). InnerVolumeSpecName "kube-api-access-l7z59". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:32:02.668586 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:02.668539    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-5402dd-86b6d5dd95-xzj89"
Oct 30 22:32:02.726698 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:02.726665    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61e905c3-97ef-4f17-aba7-0ea699720853-kube-api-access-4wxgf\" (UniqueName: \"kubernetes.io/projected/61e905c3-97ef-4f17-aba7-0ea699720853-kube-api-access-4wxgf\") pod \"61e905c3-97ef-4f17-aba7-0ea699720853\" (UID: \"61e905c3-97ef-4f17-aba7-0ea699720853\") "
Oct 30 22:32:02.726877 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:02.726856    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61e905c3-97ef-4f17-aba7-0ea699720853-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61e905c3-97ef-4f17-aba7-0ea699720853-plugins-config-volume\") pod \"61e905c3-97ef-4f17-aba7-0ea699720853\" (UID: \"61e905c3-97ef-4f17-aba7-0ea699720853\") "
Oct 30 22:32:02.727197 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:02.727176    5061 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a02d777c-062a-4182-81e7-1952fa5924f5-plugins-config-volume\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:32:02.727313 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:02.727198    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7z59\" (UniqueName: \"kubernetes.io/projected/a02d777c-062a-4182-81e7-1952fa5924f5-kube-api-access-l7z59\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:32:02.727313 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:02.727278    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61e905c3-97ef-4f17-aba7-0ea699720853-plugins-config-volume" pod "61e905c3-97ef-4f17-aba7-0ea699720853" (UID: "61e905c3-97ef-4f17-aba7-0ea699720853"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:32:02.728781 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:02.728764    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61e905c3-97ef-4f17-aba7-0ea699720853-kube-api-access-4wxgf" pod "61e905c3-97ef-4f17-aba7-0ea699720853" (UID: "61e905c3-97ef-4f17-aba7-0ea699720853"). InnerVolumeSpecName "kube-api-access-4wxgf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:32:02.828630 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:02.828594    5061 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61e905c3-97ef-4f17-aba7-0ea699720853-plugins-config-volume\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:32:02.828630 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:02.828622    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wxgf\" (UniqueName: \"kubernetes.io/projected/61e905c3-97ef-4f17-aba7-0ea699720853-kube-api-access-4wxgf\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:32:03.210339 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:03.210301    5061 generic.go:356] "Generic (PLEG): container finished" podID="a02d777c-062a-4182-81e7-1952fa5924f5" containerID="5c6bc908e527fe0e7d23233fa6a218eeddb3b58cc9984d5095231dcdee2dc100" exitCode=0
Oct 30 22:32:03.210589 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:03.210353    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-5402dd-57dc966bb-4x6rt"
Oct 30 22:32:03.210589 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:03.210391    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-5402dd-57dc966bb-4x6rt" event={"ID":"a02d777c-062a-4182-81e7-1952fa5924f5","Type":"ContainerDied","Data":"5c6bc908e527fe0e7d23233fa6a218eeddb3b58cc9984d5095231dcdee2dc100"}
Oct 30 22:32:03.210589 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:03.210447    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-5402dd-57dc966bb-4x6rt" event={"ID":"a02d777c-062a-4182-81e7-1952fa5924f5","Type":"ContainerDied","Data":"bc89d81f3bee8966fa5ea28d4bf0a6b8b712ac5f594357c9997107b84c790dc5"}
Oct 30 22:32:03.210589 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:03.210477    5061 scope.go:122] "RemoveContainer" containerID="5c6bc908e527fe0e7d23233fa6a218eeddb3b58cc9984d5095231dcdee2dc100"
Oct 30 22:32:03.211837 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:03.211816    5061 generic.go:356] "Generic (PLEG): container finished" podID="61e905c3-97ef-4f17-aba7-0ea699720853" containerID="e8e3bf7f3a42e9c86d0d52fafd396750222aa6219defa39e2431640714f71120" exitCode=0
Oct 30 22:32:03.211968 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:03.211856    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-5402dd-86b6d5dd95-xzj89"
Oct 30 22:32:03.211968 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:03.211871    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-5402dd-86b6d5dd95-xzj89" event={"ID":"61e905c3-97ef-4f17-aba7-0ea699720853","Type":"ContainerDied","Data":"e8e3bf7f3a42e9c86d0d52fafd396750222aa6219defa39e2431640714f71120"}
Oct 30 22:32:03.211968 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:03.211909    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-5402dd-86b6d5dd95-xzj89" event={"ID":"61e905c3-97ef-4f17-aba7-0ea699720853","Type":"ContainerDied","Data":"0c03759fb1f54414154446c8b6be91977853e0da3fa67d01b0a0381e1f18b691"}
Oct 30 22:32:03.221042 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:03.221018    5061 scope.go:122] "RemoveContainer" containerID="5c6bc908e527fe0e7d23233fa6a218eeddb3b58cc9984d5095231dcdee2dc100"
Oct 30 22:32:03.221335 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:32:03.221313    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c6bc908e527fe0e7d23233fa6a218eeddb3b58cc9984d5095231dcdee2dc100\": container with ID starting with 5c6bc908e527fe0e7d23233fa6a218eeddb3b58cc9984d5095231dcdee2dc100 not found: ID does not exist" containerID="5c6bc908e527fe0e7d23233fa6a218eeddb3b58cc9984d5095231dcdee2dc100"
Oct 30 22:32:03.221407 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:03.221341    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c6bc908e527fe0e7d23233fa6a218eeddb3b58cc9984d5095231dcdee2dc100"} err="failed to get container status \"5c6bc908e527fe0e7d23233fa6a218eeddb3b58cc9984d5095231dcdee2dc100\": rpc error: code = NotFound desc = could not find container \"5c6bc908e527fe0e7d23233fa6a218eeddb3b58cc9984d5095231dcdee2dc100\": container with ID starting with 5c6bc908e527fe0e7d23233fa6a218eeddb3b58cc9984d5095231dcdee2dc100 not found: ID does not exist"
Oct 30 22:32:03.221407 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:03.221358    5061 scope.go:122] "RemoveContainer" containerID="e8e3bf7f3a42e9c86d0d52fafd396750222aa6219defa39e2431640714f71120"
Oct 30 22:32:03.229462 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:03.229444    5061 scope.go:122] "RemoveContainer" containerID="e8e3bf7f3a42e9c86d0d52fafd396750222aa6219defa39e2431640714f71120"
Oct 30 22:32:03.229739 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:32:03.229719    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8e3bf7f3a42e9c86d0d52fafd396750222aa6219defa39e2431640714f71120\": container with ID starting with e8e3bf7f3a42e9c86d0d52fafd396750222aa6219defa39e2431640714f71120 not found: ID does not exist" containerID="e8e3bf7f3a42e9c86d0d52fafd396750222aa6219defa39e2431640714f71120"
Oct 30 22:32:03.229790 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:03.229745    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8e3bf7f3a42e9c86d0d52fafd396750222aa6219defa39e2431640714f71120"} err="failed to get container status \"e8e3bf7f3a42e9c86d0d52fafd396750222aa6219defa39e2431640714f71120\": rpc error: code = NotFound desc = could not find container \"e8e3bf7f3a42e9c86d0d52fafd396750222aa6219defa39e2431640714f71120\": container with ID starting with e8e3bf7f3a42e9c86d0d52fafd396750222aa6219defa39e2431640714f71120 not found: ID does not exist"
Oct 30 22:32:03.366491 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:03.366460    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-5402dd-86b6d5dd95-xzj89"]
Oct 30 22:32:03.413369 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:03.413341    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-5402dd-86b6d5dd95-xzj89"]
Oct 30 22:32:03.477013 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:03.476956    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-5402dd-57dc966bb-4x6rt"]
Oct 30 22:32:03.481728 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:03.481706    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-5402dd-57dc966bb-4x6rt"]
Oct 30 22:32:04.527539 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:04.527497    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61e905c3-97ef-4f17-aba7-0ea699720853" path="/var/lib/kubelet/pods/61e905c3-97ef-4f17-aba7-0ea699720853/volumes"
Oct 30 22:32:04.527929 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:04.527913    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a02d777c-062a-4182-81e7-1952fa5924f5" path="/var/lib/kubelet/pods/a02d777c-062a-4182-81e7-1952fa5924f5/volumes"
Oct 30 22:32:06.987331 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:06.987300    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-txl6d"]
Oct 30 22:32:06.987769 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:06.987518    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-txl6d" podUID="cbaf4a9d-78f3-449b-a2ee-3b7fe4421745" containerName="istio-proxy" containerID="cri-o://7b168911b0017899b9cdfe2f5a400ad248031af89a9397a7f55f1aa93319860f" gracePeriod=30
Oct 30 22:32:12.248463 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:12.248434    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-txl6d"
Oct 30 22:32:12.264182 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:12.264121    5061 generic.go:356] "Generic (PLEG): container finished" podID="cbaf4a9d-78f3-449b-a2ee-3b7fe4421745" containerID="7b168911b0017899b9cdfe2f5a400ad248031af89a9397a7f55f1aa93319860f" exitCode=0
Oct 30 22:32:12.264182 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:12.264164    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-txl6d"
Oct 30 22:32:12.264371 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:12.264283    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-txl6d" event={"ID":"cbaf4a9d-78f3-449b-a2ee-3b7fe4421745","Type":"ContainerDied","Data":"7b168911b0017899b9cdfe2f5a400ad248031af89a9397a7f55f1aa93319860f"}
Oct 30 22:32:12.264465 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:12.264447    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-txl6d" event={"ID":"cbaf4a9d-78f3-449b-a2ee-3b7fe4421745","Type":"ContainerDied","Data":"d807d14702d3650a9e4d907370d1c0fd829bf89174b1176ffd51cc9d74612ad4"}
Oct 30 22:32:12.264501 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:12.264485    5061 scope.go:122] "RemoveContainer" containerID="7b168911b0017899b9cdfe2f5a400ad248031af89a9397a7f55f1aa93319860f"
Oct 30 22:32:12.273112 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:12.273096    5061 scope.go:122] "RemoveContainer" containerID="7b168911b0017899b9cdfe2f5a400ad248031af89a9397a7f55f1aa93319860f"
Oct 30 22:32:12.273391 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:32:12.273367    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b168911b0017899b9cdfe2f5a400ad248031af89a9397a7f55f1aa93319860f\": container with ID starting with 7b168911b0017899b9cdfe2f5a400ad248031af89a9397a7f55f1aa93319860f not found: ID does not exist" containerID="7b168911b0017899b9cdfe2f5a400ad248031af89a9397a7f55f1aa93319860f"
Oct 30 22:32:12.273439 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:12.273403    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b168911b0017899b9cdfe2f5a400ad248031af89a9397a7f55f1aa93319860f"} err="failed to get container status \"7b168911b0017899b9cdfe2f5a400ad248031af89a9397a7f55f1aa93319860f\": rpc error: code = NotFound desc = could not find container \"7b168911b0017899b9cdfe2f5a400ad248031af89a9397a7f55f1aa93319860f\": container with ID starting with 7b168911b0017899b9cdfe2f5a400ad248031af89a9397a7f55f1aa93319860f not found: ID does not exist"
Oct 30 22:32:12.327318 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:12.327292    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbaf4a9d-78f3-449b-a2ee-3b7fe4421745-kube-api-access-b7hxv\" (UniqueName: \"kubernetes.io/projected/cbaf4a9d-78f3-449b-a2ee-3b7fe4421745-kube-api-access-b7hxv\") pod \"cbaf4a9d-78f3-449b-a2ee-3b7fe4421745\" (UID: \"cbaf4a9d-78f3-449b-a2ee-3b7fe4421745\") "
Oct 30 22:32:12.327432 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:12.327338    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbaf4a9d-78f3-449b-a2ee-3b7fe4421745-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbaf4a9d-78f3-449b-a2ee-3b7fe4421745-istiod-ca-cert\") pod \"cbaf4a9d-78f3-449b-a2ee-3b7fe4421745\" (UID: \"cbaf4a9d-78f3-449b-a2ee-3b7fe4421745\") "
Oct 30 22:32:12.327432 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:12.327385    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbaf4a9d-78f3-449b-a2ee-3b7fe4421745-istio-token\" (UniqueName: \"kubernetes.io/projected/cbaf4a9d-78f3-449b-a2ee-3b7fe4421745-istio-token\") pod \"cbaf4a9d-78f3-449b-a2ee-3b7fe4421745\" (UID: \"cbaf4a9d-78f3-449b-a2ee-3b7fe4421745\") "
Oct 30 22:32:12.327432 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:12.327408    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbaf4a9d-78f3-449b-a2ee-3b7fe4421745-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbaf4a9d-78f3-449b-a2ee-3b7fe4421745-workload-certs\") pod \"cbaf4a9d-78f3-449b-a2ee-3b7fe4421745\" (UID: \"cbaf4a9d-78f3-449b-a2ee-3b7fe4421745\") "
Oct 30 22:32:12.327432 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:12.327429    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbaf4a9d-78f3-449b-a2ee-3b7fe4421745-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbaf4a9d-78f3-449b-a2ee-3b7fe4421745-credential-socket\") pod \"cbaf4a9d-78f3-449b-a2ee-3b7fe4421745\" (UID: \"cbaf4a9d-78f3-449b-a2ee-3b7fe4421745\") "
Oct 30 22:32:12.327697 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:12.327467    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cbaf4a9d-78f3-449b-a2ee-3b7fe4421745-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbaf4a9d-78f3-449b-a2ee-3b7fe4421745-istio-podinfo\") pod \"cbaf4a9d-78f3-449b-a2ee-3b7fe4421745\" (UID: \"cbaf4a9d-78f3-449b-a2ee-3b7fe4421745\") "
Oct 30 22:32:12.327697 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:12.327484    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbaf4a9d-78f3-449b-a2ee-3b7fe4421745-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbaf4a9d-78f3-449b-a2ee-3b7fe4421745-istio-data\") pod \"cbaf4a9d-78f3-449b-a2ee-3b7fe4421745\" (UID: \"cbaf4a9d-78f3-449b-a2ee-3b7fe4421745\") "
Oct 30 22:32:12.327697 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:12.327511    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbaf4a9d-78f3-449b-a2ee-3b7fe4421745-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbaf4a9d-78f3-449b-a2ee-3b7fe4421745-workload-socket\") pod \"cbaf4a9d-78f3-449b-a2ee-3b7fe4421745\" (UID: \"cbaf4a9d-78f3-449b-a2ee-3b7fe4421745\") "
Oct 30 22:32:12.327697 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:12.327609    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbaf4a9d-78f3-449b-a2ee-3b7fe4421745-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbaf4a9d-78f3-449b-a2ee-3b7fe4421745-istio-envoy\") pod \"cbaf4a9d-78f3-449b-a2ee-3b7fe4421745\" (UID: \"cbaf4a9d-78f3-449b-a2ee-3b7fe4421745\") "
Oct 30 22:32:12.327924 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:12.327756    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbaf4a9d-78f3-449b-a2ee-3b7fe4421745-workload-certs" pod "cbaf4a9d-78f3-449b-a2ee-3b7fe4421745" (UID: "cbaf4a9d-78f3-449b-a2ee-3b7fe4421745"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:32:12.327924 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:12.327847    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbaf4a9d-78f3-449b-a2ee-3b7fe4421745-istio-data" pod "cbaf4a9d-78f3-449b-a2ee-3b7fe4421745" (UID: "cbaf4a9d-78f3-449b-a2ee-3b7fe4421745"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:32:12.327924 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:12.327855    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbaf4a9d-78f3-449b-a2ee-3b7fe4421745-istiod-ca-cert" pod "cbaf4a9d-78f3-449b-a2ee-3b7fe4421745" (UID: "cbaf4a9d-78f3-449b-a2ee-3b7fe4421745"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:32:12.328091 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:12.328007    5061 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbaf4a9d-78f3-449b-a2ee-3b7fe4421745-istiod-ca-cert\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:32:12.328091 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:12.328029    5061 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbaf4a9d-78f3-449b-a2ee-3b7fe4421745-workload-certs\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:32:12.328091 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:12.328043    5061 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbaf4a9d-78f3-449b-a2ee-3b7fe4421745-istio-data\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:32:12.328091 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:12.328056    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbaf4a9d-78f3-449b-a2ee-3b7fe4421745-credential-socket" pod "cbaf4a9d-78f3-449b-a2ee-3b7fe4421745" (UID: "cbaf4a9d-78f3-449b-a2ee-3b7fe4421745"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:32:12.328289 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:12.328107    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbaf4a9d-78f3-449b-a2ee-3b7fe4421745-workload-socket" pod "cbaf4a9d-78f3-449b-a2ee-3b7fe4421745" (UID: "cbaf4a9d-78f3-449b-a2ee-3b7fe4421745"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:32:12.329563 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:12.329514    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbaf4a9d-78f3-449b-a2ee-3b7fe4421745-kube-api-access-b7hxv" pod "cbaf4a9d-78f3-449b-a2ee-3b7fe4421745" (UID: "cbaf4a9d-78f3-449b-a2ee-3b7fe4421745"). InnerVolumeSpecName "kube-api-access-b7hxv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:32:12.329805 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:12.329784    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbaf4a9d-78f3-449b-a2ee-3b7fe4421745-istio-envoy" pod "cbaf4a9d-78f3-449b-a2ee-3b7fe4421745" (UID: "cbaf4a9d-78f3-449b-a2ee-3b7fe4421745"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:32:12.330063 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:12.330043    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cbaf4a9d-78f3-449b-a2ee-3b7fe4421745-istio-podinfo" pod "cbaf4a9d-78f3-449b-a2ee-3b7fe4421745" (UID: "cbaf4a9d-78f3-449b-a2ee-3b7fe4421745"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:32:12.330152 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:12.330046    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbaf4a9d-78f3-449b-a2ee-3b7fe4421745-istio-token" pod "cbaf4a9d-78f3-449b-a2ee-3b7fe4421745" (UID: "cbaf4a9d-78f3-449b-a2ee-3b7fe4421745"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:32:12.429005 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:12.428973    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7hxv\" (UniqueName: \"kubernetes.io/projected/cbaf4a9d-78f3-449b-a2ee-3b7fe4421745-kube-api-access-b7hxv\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:32:12.429005 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:12.429003    5061 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbaf4a9d-78f3-449b-a2ee-3b7fe4421745-istio-token\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:32:12.429194 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:12.429013    5061 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbaf4a9d-78f3-449b-a2ee-3b7fe4421745-credential-socket\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:32:12.429194 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:12.429028    5061 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbaf4a9d-78f3-449b-a2ee-3b7fe4421745-istio-podinfo\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:32:12.429194 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:12.429040    5061 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbaf4a9d-78f3-449b-a2ee-3b7fe4421745-workload-socket\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:32:12.429194 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:12.429048    5061 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbaf4a9d-78f3-449b-a2ee-3b7fe4421745-istio-envoy\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:32:12.674925 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:12.674891    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-txl6d"]
Oct 30 22:32:12.680226 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:12.680188    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-txl6d"]
Oct 30 22:32:14.532070 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:14.532027    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbaf4a9d-78f3-449b-a2ee-3b7fe4421745" path="/var/lib/kubelet/pods/cbaf4a9d-78f3-449b-a2ee-3b7fe4421745/volumes"
Oct 30 22:32:24.666378 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:32:24.666322    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 22:32:26.961230 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:26.961192    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-sclrt_5e74071e-0462-42b5-a05c-d90bb638478a/console-operator/2.log"
Oct 30 22:32:27.004226 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:27.004202    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-sclrt_5e74071e-0462-42b5-a05c-d90bb638478a/console-operator/2.log"
Oct 30 22:32:29.453116 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:29.453081    5061 scope.go:122] "RemoveContainer" containerID="b184ee3e93d46255d09ef818e3c251e1de74d3af184c436e4ee449355b3802ab"
Oct 30 22:32:41.044022 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:41.043941    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84b1935f-m69v4"]
Oct 30 22:32:41.091322 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:41.091290    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84b1935f-m69v4"]
Oct 30 22:32:41.093167 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:41.093141    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31c8b66a-n4cx4"]
Oct 30 22:32:41.094877 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:41.094854    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31c8b66a-n4cx4"]
Oct 30 22:32:42.527782 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:42.527748    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44d23433-b77d-49d4-a5a3-a969f2d0507d" path="/var/lib/kubelet/pods/44d23433-b77d-49d4-a5a3-a969f2d0507d/volumes"
Oct 30 22:32:42.528195 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:42.528165    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fc81a52-bb17-4dc5-9de2-62ae34ee7917" path="/var/lib/kubelet/pods/9fc81a52-bb17-4dc5-9de2-62ae34ee7917/volumes"
Oct 30 22:32:46.307911 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:46.307861    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-6hts8"]
Oct 30 22:32:46.308299 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:46.308257    5061 cpu_manager.go:414] "containerMap: removing container" podUID="2caef524-f1b2-4356-a253-66a13d80d8ca" containerName="revoke-keys"
Oct 30 22:32:46.308299 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:46.308281    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2caef524-f1b2-4356-a253-66a13d80d8ca" containerName="revoke-keys"
Oct 30 22:32:46.308299 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:46.308293    5061 cpu_manager.go:414] "containerMap: removing container" podUID="cbaf4a9d-78f3-449b-a2ee-3b7fe4421745" containerName="istio-proxy"
Oct 30 22:32:46.308299 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:46.308298    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbaf4a9d-78f3-449b-a2ee-3b7fe4421745" containerName="istio-proxy"
Oct 30 22:32:46.308442 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:46.308314    5061 cpu_manager.go:414] "containerMap: removing container" podUID="a02d777c-062a-4182-81e7-1952fa5924f5" containerName="payload-processing"
Oct 30 22:32:46.308442 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:46.308319    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a02d777c-062a-4182-81e7-1952fa5924f5" containerName="payload-processing"
Oct 30 22:32:46.308442 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:46.308326    5061 cpu_manager.go:414] "containerMap: removing container" podUID="61e905c3-97ef-4f17-aba7-0ea699720853" containerName="payload-pre-processing"
Oct 30 22:32:46.308442 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:46.308331    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61e905c3-97ef-4f17-aba7-0ea699720853" containerName="payload-pre-processing"
Oct 30 22:32:46.308442 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:46.308404    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="2caef524-f1b2-4356-a253-66a13d80d8ca" containerName="revoke-keys"
Oct 30 22:32:46.308442 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:46.308413    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbaf4a9d-78f3-449b-a2ee-3b7fe4421745" containerName="istio-proxy"
Oct 30 22:32:46.308442 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:46.308423    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="a02d777c-062a-4182-81e7-1952fa5924f5" containerName="payload-processing"
Oct 30 22:32:46.308442 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:46.308429    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="61e905c3-97ef-4f17-aba7-0ea699720853" containerName="payload-pre-processing"
Oct 30 22:32:46.313084 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:46.313063    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-6hts8"
Oct 30 22:32:46.362400 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:46.362370    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:32:46.409337 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:46.409299    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:32:46.409492 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:46.409466    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:32:46.409588 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:46.409569    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gdmfg\""
Oct 30 22:32:46.411479 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:46.411456    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-6hts8"]
Oct 30 22:32:46.447437 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:46.447410    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pvjp\" (UniqueName: \"kubernetes.io/projected/f5eb3cbe-605a-44a9-b51e-a2da5ab313b8-kube-api-access-5pvjp\") pod \"maas-api-revoke-sub-85237e45-6hts8\" (UID: \"f5eb3cbe-605a-44a9-b51e-a2da5ab313b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-6hts8"
Oct 30 22:32:46.447582 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:46.447475    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5eb3cbe-605a-44a9-b51e-a2da5ab313b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-6hts8\" (UID: \"f5eb3cbe-605a-44a9-b51e-a2da5ab313b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-6hts8"
Oct 30 22:32:46.447635 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:46.447607    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5eb3cbe-605a-44a9-b51e-a2da5ab313b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-6hts8\" (UID: \"f5eb3cbe-605a-44a9-b51e-a2da5ab313b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-6hts8"
Oct 30 22:32:46.548340 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:46.548312    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5eb3cbe-605a-44a9-b51e-a2da5ab313b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-6hts8\" (UID: \"f5eb3cbe-605a-44a9-b51e-a2da5ab313b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-6hts8"
Oct 30 22:32:46.548559 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:46.548374    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5eb3cbe-605a-44a9-b51e-a2da5ab313b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-6hts8\" (UID: \"f5eb3cbe-605a-44a9-b51e-a2da5ab313b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-6hts8"
Oct 30 22:32:46.548559 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:46.548414    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pvjp\" (UniqueName: \"kubernetes.io/projected/f5eb3cbe-605a-44a9-b51e-a2da5ab313b8-kube-api-access-5pvjp\") pod \"maas-api-revoke-sub-85237e45-6hts8\" (UID: \"f5eb3cbe-605a-44a9-b51e-a2da5ab313b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-6hts8"
Oct 30 22:32:46.549140 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:46.549114    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5eb3cbe-605a-44a9-b51e-a2da5ab313b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-6hts8\" (UID: \"f5eb3cbe-605a-44a9-b51e-a2da5ab313b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-6hts8"
Oct 30 22:32:46.550745 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:46.550724    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5eb3cbe-605a-44a9-b51e-a2da5ab313b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-6hts8\" (UID: \"f5eb3cbe-605a-44a9-b51e-a2da5ab313b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-6hts8"
Oct 30 22:32:46.602875 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:46.602809    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pvjp\" (UniqueName: \"kubernetes.io/projected/f5eb3cbe-605a-44a9-b51e-a2da5ab313b8-kube-api-access-5pvjp\") pod \"maas-api-revoke-sub-85237e45-6hts8\" (UID: \"f5eb3cbe-605a-44a9-b51e-a2da5ab313b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-6hts8"
Oct 30 22:32:46.625689 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:46.625663    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-6hts8"
Oct 30 22:32:46.987278 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:46.987256    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-6hts8"]
Oct 30 22:32:46.989503 ip-10-0-130-60 kubenswrapper[5061]: W1030 22:32:46.989475    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5eb3cbe_605a_44a9_b51e_a2da5ab313b8.slice/crio-243ffcc4ef7a01e5d8650225210ca21f547977910a0e97e6e9de57045c45ffdb WatchSource:0}: Error finding container 243ffcc4ef7a01e5d8650225210ca21f547977910a0e97e6e9de57045c45ffdb: Status 404 returned error can't find the container with id 243ffcc4ef7a01e5d8650225210ca21f547977910a0e97e6e9de57045c45ffdb
Oct 30 22:32:47.453366 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:47.453325    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-6hts8" event={"ID":"f5eb3cbe-605a-44a9-b51e-a2da5ab313b8","Type":"ContainerStarted","Data":"df54277ebcb9e9926995483481e503296b08a16e04990e14b3366138273f41ee"}
Oct 30 22:32:47.453823 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:47.453385    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-6hts8" event={"ID":"f5eb3cbe-605a-44a9-b51e-a2da5ab313b8","Type":"ContainerStarted","Data":"243ffcc4ef7a01e5d8650225210ca21f547977910a0e97e6e9de57045c45ffdb"}
Oct 30 22:32:47.562352 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:47.562293    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-6hts8" podStartSLOduration=1.5622799330000001 podStartE2EDuration="1.562279933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:32:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:32:47.561264043 +0000 UTC m=+3021.174033812" watchObservedRunningTime="2026-10-30 22:32:47.562279933 +0000 UTC m=+3021.175049690"
Oct 30 22:32:48.459815 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:48.459779    5061 generic.go:356] "Generic (PLEG): container finished" podID="f5eb3cbe-605a-44a9-b51e-a2da5ab313b8" containerID="df54277ebcb9e9926995483481e503296b08a16e04990e14b3366138273f41ee" exitCode=0
Oct 30 22:32:48.460296 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:48.459868    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-6hts8" event={"ID":"f5eb3cbe-605a-44a9-b51e-a2da5ab313b8","Type":"ContainerDied","Data":"df54277ebcb9e9926995483481e503296b08a16e04990e14b3366138273f41ee"}
Oct 30 22:32:49.602877 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:49.602848    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-6hts8"
Oct 30 22:32:49.680088 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:49.680057    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f5eb3cbe-605a-44a9-b51e-a2da5ab313b8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5eb3cbe-605a-44a9-b51e-a2da5ab313b8-maas-api-xks-ca\") pod \"f5eb3cbe-605a-44a9-b51e-a2da5ab313b8\" (UID: \"f5eb3cbe-605a-44a9-b51e-a2da5ab313b8\") "
Oct 30 22:32:49.680229 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:49.680109    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5eb3cbe-605a-44a9-b51e-a2da5ab313b8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5eb3cbe-605a-44a9-b51e-a2da5ab313b8-maas-api-openshift-ca\") pod \"f5eb3cbe-605a-44a9-b51e-a2da5ab313b8\" (UID: \"f5eb3cbe-605a-44a9-b51e-a2da5ab313b8\") "
Oct 30 22:32:49.680229 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:49.680188    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5eb3cbe-605a-44a9-b51e-a2da5ab313b8-kube-api-access-5pvjp\" (UniqueName: \"kubernetes.io/projected/f5eb3cbe-605a-44a9-b51e-a2da5ab313b8-kube-api-access-5pvjp\") pod \"f5eb3cbe-605a-44a9-b51e-a2da5ab313b8\" (UID: \"f5eb3cbe-605a-44a9-b51e-a2da5ab313b8\") "
Oct 30 22:32:49.680585 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:49.680543    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5eb3cbe-605a-44a9-b51e-a2da5ab313b8-maas-api-openshift-ca" pod "f5eb3cbe-605a-44a9-b51e-a2da5ab313b8" (UID: "f5eb3cbe-605a-44a9-b51e-a2da5ab313b8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:32:49.682133 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:49.682108    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5eb3cbe-605a-44a9-b51e-a2da5ab313b8-maas-api-xks-ca" pod "f5eb3cbe-605a-44a9-b51e-a2da5ab313b8" (UID: "f5eb3cbe-605a-44a9-b51e-a2da5ab313b8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:32:49.682225 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:49.682203    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5eb3cbe-605a-44a9-b51e-a2da5ab313b8-kube-api-access-5pvjp" pod "f5eb3cbe-605a-44a9-b51e-a2da5ab313b8" (UID: "f5eb3cbe-605a-44a9-b51e-a2da5ab313b8"). InnerVolumeSpecName "kube-api-access-5pvjp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:32:49.781911 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:49.781829    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5eb3cbe-605a-44a9-b51e-a2da5ab313b8-maas-api-openshift-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:32:49.781911 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:49.781861    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pvjp\" (UniqueName: \"kubernetes.io/projected/f5eb3cbe-605a-44a9-b51e-a2da5ab313b8-kube-api-access-5pvjp\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:32:49.781911 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:49.781875    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5eb3cbe-605a-44a9-b51e-a2da5ab313b8-maas-api-xks-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:32:50.472789 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:50.472752    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-6hts8"
Oct 30 22:32:50.472789 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:50.472770    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-6hts8" event={"ID":"f5eb3cbe-605a-44a9-b51e-a2da5ab313b8","Type":"ContainerDied","Data":"243ffcc4ef7a01e5d8650225210ca21f547977910a0e97e6e9de57045c45ffdb"}
Oct 30 22:32:50.473021 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:50.472813    5061 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="243ffcc4ef7a01e5d8650225210ca21f547977910a0e97e6e9de57045c45ffdb"
Oct 30 22:32:51.283203 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:51.283167    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8pzt2"]
Oct 30 22:32:51.283639 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:51.283612    5061 cpu_manager.go:414] "containerMap: removing container" podUID="f5eb3cbe-605a-44a9-b51e-a2da5ab313b8" containerName="revoke-keys"
Oct 30 22:32:51.283639 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:51.283623    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5eb3cbe-605a-44a9-b51e-a2da5ab313b8" containerName="revoke-keys"
Oct 30 22:32:51.283728 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:51.283685    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5eb3cbe-605a-44a9-b51e-a2da5ab313b8" containerName="revoke-keys"
Oct 30 22:32:51.287117 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:51.287097    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8pzt2"
Oct 30 22:32:51.395112 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:51.395076    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8pzt2"]
Oct 30 22:32:51.396966 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:51.396928    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3808649-2258-4508-be89-8e0c57ca83e2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-8pzt2\" (UID: \"b3808649-2258-4508-be89-8e0c57ca83e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8pzt2"
Oct 30 22:32:51.397205 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:51.397175    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3808649-2258-4508-be89-8e0c57ca83e2-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-8pzt2\" (UID: \"b3808649-2258-4508-be89-8e0c57ca83e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8pzt2"
Oct 30 22:32:51.397325 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:51.397289    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3808649-2258-4508-be89-8e0c57ca83e2-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-8pzt2\" (UID: \"b3808649-2258-4508-be89-8e0c57ca83e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8pzt2"
Oct 30 22:32:51.397397 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:51.397339    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3808649-2258-4508-be89-8e0c57ca83e2-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-8pzt2\" (UID: \"b3808649-2258-4508-be89-8e0c57ca83e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8pzt2"
Oct 30 22:32:51.397397 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:51.397360    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3808649-2258-4508-be89-8e0c57ca83e2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-8pzt2\" (UID: \"b3808649-2258-4508-be89-8e0c57ca83e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8pzt2"
Oct 30 22:32:51.397590 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:51.397431    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3808649-2258-4508-be89-8e0c57ca83e2-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-8pzt2\" (UID: \"b3808649-2258-4508-be89-8e0c57ca83e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8pzt2"
Oct 30 22:32:51.397590 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:51.397490    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3808649-2258-4508-be89-8e0c57ca83e2-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-8pzt2\" (UID: \"b3808649-2258-4508-be89-8e0c57ca83e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8pzt2"
Oct 30 22:32:51.397724 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:51.397597    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxq5z\" (UniqueName: \"kubernetes.io/projected/b3808649-2258-4508-be89-8e0c57ca83e2-kube-api-access-dxq5z\") pod \"maas-default-gateway-openshift-default-694db55c87-8pzt2\" (UID: \"b3808649-2258-4508-be89-8e0c57ca83e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8pzt2"
Oct 30 22:32:51.397724 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:51.397648    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3808649-2258-4508-be89-8e0c57ca83e2-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-8pzt2\" (UID: \"b3808649-2258-4508-be89-8e0c57ca83e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8pzt2"
Oct 30 22:32:51.498915 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:51.498880    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3808649-2258-4508-be89-8e0c57ca83e2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-8pzt2\" (UID: \"b3808649-2258-4508-be89-8e0c57ca83e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8pzt2"
Oct 30 22:32:51.499106 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:51.498920    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3808649-2258-4508-be89-8e0c57ca83e2-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-8pzt2\" (UID: \"b3808649-2258-4508-be89-8e0c57ca83e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8pzt2"
Oct 30 22:32:51.499106 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:51.498948    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3808649-2258-4508-be89-8e0c57ca83e2-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-8pzt2\" (UID: \"b3808649-2258-4508-be89-8e0c57ca83e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8pzt2"
Oct 30 22:32:51.499106 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:51.498982    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3808649-2258-4508-be89-8e0c57ca83e2-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-8pzt2\" (UID: \"b3808649-2258-4508-be89-8e0c57ca83e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8pzt2"
Oct 30 22:32:51.499106 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:51.499001    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3808649-2258-4508-be89-8e0c57ca83e2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-8pzt2\" (UID: \"b3808649-2258-4508-be89-8e0c57ca83e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8pzt2"
Oct 30 22:32:51.499106 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:51.499018    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3808649-2258-4508-be89-8e0c57ca83e2-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-8pzt2\" (UID: \"b3808649-2258-4508-be89-8e0c57ca83e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8pzt2"
Oct 30 22:32:51.499106 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:51.499033    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3808649-2258-4508-be89-8e0c57ca83e2-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-8pzt2\" (UID: \"b3808649-2258-4508-be89-8e0c57ca83e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8pzt2"
Oct 30 22:32:51.499106 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:51.499056    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxq5z\" (UniqueName: \"kubernetes.io/projected/b3808649-2258-4508-be89-8e0c57ca83e2-kube-api-access-dxq5z\") pod \"maas-default-gateway-openshift-default-694db55c87-8pzt2\" (UID: \"b3808649-2258-4508-be89-8e0c57ca83e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8pzt2"
Oct 30 22:32:51.499601 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:51.499172    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3808649-2258-4508-be89-8e0c57ca83e2-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-8pzt2\" (UID: \"b3808649-2258-4508-be89-8e0c57ca83e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8pzt2"
Oct 30 22:32:51.499601 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:51.499428    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3808649-2258-4508-be89-8e0c57ca83e2-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-8pzt2\" (UID: \"b3808649-2258-4508-be89-8e0c57ca83e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8pzt2"
Oct 30 22:32:51.499601 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:51.499508    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3808649-2258-4508-be89-8e0c57ca83e2-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-8pzt2\" (UID: \"b3808649-2258-4508-be89-8e0c57ca83e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8pzt2"
Oct 30 22:32:51.499794 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:51.499640    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3808649-2258-4508-be89-8e0c57ca83e2-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-8pzt2\" (UID: \"b3808649-2258-4508-be89-8e0c57ca83e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8pzt2"
Oct 30 22:32:51.499794 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:51.499737    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3808649-2258-4508-be89-8e0c57ca83e2-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-8pzt2\" (UID: \"b3808649-2258-4508-be89-8e0c57ca83e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8pzt2"
Oct 30 22:32:51.499905 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:51.499861    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3808649-2258-4508-be89-8e0c57ca83e2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-8pzt2\" (UID: \"b3808649-2258-4508-be89-8e0c57ca83e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8pzt2"
Oct 30 22:32:51.501387 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:51.501362    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3808649-2258-4508-be89-8e0c57ca83e2-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-8pzt2\" (UID: \"b3808649-2258-4508-be89-8e0c57ca83e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8pzt2"
Oct 30 22:32:51.501522 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:51.501502    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3808649-2258-4508-be89-8e0c57ca83e2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-8pzt2\" (UID: \"b3808649-2258-4508-be89-8e0c57ca83e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8pzt2"
Oct 30 22:32:51.553733 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:51.553698    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3808649-2258-4508-be89-8e0c57ca83e2-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-8pzt2\" (UID: \"b3808649-2258-4508-be89-8e0c57ca83e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8pzt2"
Oct 30 22:32:51.553876 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:51.553809    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxq5z\" (UniqueName: \"kubernetes.io/projected/b3808649-2258-4508-be89-8e0c57ca83e2-kube-api-access-dxq5z\") pod \"maas-default-gateway-openshift-default-694db55c87-8pzt2\" (UID: \"b3808649-2258-4508-be89-8e0c57ca83e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8pzt2"
Oct 30 22:32:51.600302 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:51.600272    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8pzt2"
Oct 30 22:32:52.015277 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:52.015246    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8pzt2"]
Oct 30 22:32:52.016233 ip-10-0-130-60 kubenswrapper[5061]: W1030 22:32:52.016196    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3808649_2258_4508_be89_8e0c57ca83e2.slice/crio-db967cdff6f0f3b1b8a0b2bed707dd85547e6f907153aa5cc97dcf3c0a825052 WatchSource:0}: Error finding container db967cdff6f0f3b1b8a0b2bed707dd85547e6f907153aa5cc97dcf3c0a825052: Status 404 returned error can't find the container with id db967cdff6f0f3b1b8a0b2bed707dd85547e6f907153aa5cc97dcf3c0a825052
Oct 30 22:32:52.018721 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:52.018686    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:32:52.018821 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:52.018753    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:32:52.018821 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:52.018787    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:32:52.489577 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:52.489310    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8pzt2" event={"ID":"b3808649-2258-4508-be89-8e0c57ca83e2","Type":"ContainerStarted","Data":"6764940b91e932d1996fa8e64066329500d8cda8d2c3fce7749e84c2fcc35910"}
Oct 30 22:32:52.489577 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:52.489399    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8pzt2" event={"ID":"b3808649-2258-4508-be89-8e0c57ca83e2","Type":"ContainerStarted","Data":"db967cdff6f0f3b1b8a0b2bed707dd85547e6f907153aa5cc97dcf3c0a825052"}
Oct 30 22:32:52.600624 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:52.600584    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8pzt2"
Oct 30 22:32:52.647793 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:52.647733    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8pzt2" podStartSLOduration=1.647717277 podStartE2EDuration="1.647717277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:32:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:32:52.646664937 +0000 UTC m=+3026.259434714" watchObservedRunningTime="2026-10-30 22:32:52.647717277 +0000 UTC m=+3026.260487035"
Oct 30 22:32:53.602842 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:53.602800    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-8pzt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body=
Oct 30 22:32:53.603344 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:53.602870    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8pzt2" podUID="b3808649-2258-4508-be89-8e0c57ca83e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused"
Oct 30 22:32:54.601482 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:54.601446    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-8pzt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body=
Oct 30 22:32:54.601697 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:54.601508    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8pzt2" podUID="b3808649-2258-4508-be89-8e0c57ca83e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused"
Oct 30 22:32:54.666107 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:32:54.666074    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 22:32:55.600880 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:55.600843    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-8pzt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body=
Oct 30 22:32:55.601094 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:55.600898    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8pzt2" podUID="b3808649-2258-4508-be89-8e0c57ca83e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused"
Oct 30 22:32:56.601667 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:56.601631    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-8pzt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body=
Oct 30 22:32:56.602062 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:56.601686    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8pzt2" podUID="b3808649-2258-4508-be89-8e0c57ca83e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused"
Oct 30 22:32:57.600749 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:57.600713    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-8pzt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body=
Oct 30 22:32:57.600960 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:57.600770    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8pzt2" podUID="b3808649-2258-4508-be89-8e0c57ca83e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused"
Oct 30 22:32:58.601505 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:58.601469    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-8pzt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body=
Oct 30 22:32:58.602013 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:32:58.601519    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8pzt2" podUID="b3808649-2258-4508-be89-8e0c57ca83e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused"
Oct 30 22:33:00.255093 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:33:00.255054    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8pzt2"
Oct 30 22:33:00.255579 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:33:00.255554    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8pzt2"
Oct 30 22:33:00.256509 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:33:00.256483    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8pzt2"
Oct 30 22:33:24.667099 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:33:24.667061    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 22:33:29.488754 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:33:29.488721    5061 scope.go:122] "RemoveContainer" containerID="3bababed699bab326bdbf355801c461dfd5f6fc3046bbb4a5b4cff5327a8f623"
Oct 30 22:33:45.046473 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:33:45.046428    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f028c46f-tpj6q"]
Oct 30 22:33:45.090666 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:33:45.090630    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f028c46f-tpj6q"]
Oct 30 22:33:46.527601 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:33:46.527567    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f4cd395-f0a0-49dc-9deb-4eb3b5d2c67f" path="/var/lib/kubelet/pods/4f4cd395-f0a0-49dc-9deb-4eb3b5d2c67f/volumes"
Oct 30 22:33:54.666432 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:33:54.666386    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 22:33:55.082523 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:33:55.082490    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-nvm5h"]
Oct 30 22:33:55.087481 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:33:55.087459    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-nvm5h"
Oct 30 22:33:55.137170 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:33:55.137144    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:33:55.153845 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:33:55.153811    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkxt8\" (UniqueName: \"kubernetes.io/projected/c3ca1201-6968-4e3f-8eb0-4c667cc7c44c-kube-api-access-tkxt8\") pod \"maas-api-revoke-sub-c1e271a9-nvm5h\" (UID: \"c3ca1201-6968-4e3f-8eb0-4c667cc7c44c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-nvm5h"
Oct 30 22:33:55.153996 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:33:55.153973    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3ca1201-6968-4e3f-8eb0-4c667cc7c44c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-nvm5h\" (UID: \"c3ca1201-6968-4e3f-8eb0-4c667cc7c44c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-nvm5h"
Oct 30 22:33:55.154067 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:33:55.154017    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3ca1201-6968-4e3f-8eb0-4c667cc7c44c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-nvm5h\" (UID: \"c3ca1201-6968-4e3f-8eb0-4c667cc7c44c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-nvm5h"
Oct 30 22:33:55.184227 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:33:55.184200    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:33:55.184227 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:33:55.184219    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:33:55.184412 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:33:55.184197    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gdmfg\""
Oct 30 22:33:55.186291 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:33:55.186270    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-nvm5h"]
Oct 30 22:33:55.255207 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:33:55.255177    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkxt8\" (UniqueName: \"kubernetes.io/projected/c3ca1201-6968-4e3f-8eb0-4c667cc7c44c-kube-api-access-tkxt8\") pod \"maas-api-revoke-sub-c1e271a9-nvm5h\" (UID: \"c3ca1201-6968-4e3f-8eb0-4c667cc7c44c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-nvm5h"
Oct 30 22:33:55.255399 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:33:55.255242    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3ca1201-6968-4e3f-8eb0-4c667cc7c44c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-nvm5h\" (UID: \"c3ca1201-6968-4e3f-8eb0-4c667cc7c44c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-nvm5h"
Oct 30 22:33:55.255399 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:33:55.255264    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3ca1201-6968-4e3f-8eb0-4c667cc7c44c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-nvm5h\" (UID: \"c3ca1201-6968-4e3f-8eb0-4c667cc7c44c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-nvm5h"
Oct 30 22:33:55.255956 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:33:55.255934    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3ca1201-6968-4e3f-8eb0-4c667cc7c44c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-nvm5h\" (UID: \"c3ca1201-6968-4e3f-8eb0-4c667cc7c44c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-nvm5h"
Oct 30 22:33:55.257657 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:33:55.257641    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3ca1201-6968-4e3f-8eb0-4c667cc7c44c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-nvm5h\" (UID: \"c3ca1201-6968-4e3f-8eb0-4c667cc7c44c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-nvm5h"
Oct 30 22:33:55.310933 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:33:55.310899    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkxt8\" (UniqueName: \"kubernetes.io/projected/c3ca1201-6968-4e3f-8eb0-4c667cc7c44c-kube-api-access-tkxt8\") pod \"maas-api-revoke-sub-c1e271a9-nvm5h\" (UID: \"c3ca1201-6968-4e3f-8eb0-4c667cc7c44c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-nvm5h"
Oct 30 22:33:55.399683 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:33:55.399601    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-nvm5h"
Oct 30 22:33:55.771853 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:33:55.771822    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-nvm5h"]
Oct 30 22:33:55.773006 ip-10-0-130-60 kubenswrapper[5061]: W1030 22:33:55.772980    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3ca1201_6968_4e3f_8eb0_4c667cc7c44c.slice/crio-a6d004ce01ab310409942c8d6e9538c614d6a9c8ba7d88e01f58a4a2ecc58c73 WatchSource:0}: Error finding container a6d004ce01ab310409942c8d6e9538c614d6a9c8ba7d88e01f58a4a2ecc58c73: Status 404 returned error can't find the container with id a6d004ce01ab310409942c8d6e9538c614d6a9c8ba7d88e01f58a4a2ecc58c73
Oct 30 22:33:55.875910 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:33:55.875874    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-nvm5h" event={"ID":"c3ca1201-6968-4e3f-8eb0-4c667cc7c44c","Type":"ContainerStarted","Data":"a6d004ce01ab310409942c8d6e9538c614d6a9c8ba7d88e01f58a4a2ecc58c73"}
Oct 30 22:33:56.882913 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:33:56.882873    5061 generic.go:356] "Generic (PLEG): container finished" podID="c3ca1201-6968-4e3f-8eb0-4c667cc7c44c" containerID="66bf3e4b823385b1ec2ae8ca6c5c9e434aa2d67eaf59525a3f3eb545260e2b48" exitCode=0
Oct 30 22:33:56.883339 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:33:56.882953    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-nvm5h" event={"ID":"c3ca1201-6968-4e3f-8eb0-4c667cc7c44c","Type":"ContainerDied","Data":"66bf3e4b823385b1ec2ae8ca6c5c9e434aa2d67eaf59525a3f3eb545260e2b48"}
Oct 30 22:33:58.031098 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:33:58.031072    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-nvm5h"
Oct 30 22:33:58.045160 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:33:58.045135    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89af4e0a-blf2f"]
Oct 30 22:33:58.083682 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:33:58.083645    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3ca1201-6968-4e3f-8eb0-4c667cc7c44c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3ca1201-6968-4e3f-8eb0-4c667cc7c44c-maas-api-openshift-ca\") pod \"c3ca1201-6968-4e3f-8eb0-4c667cc7c44c\" (UID: \"c3ca1201-6968-4e3f-8eb0-4c667cc7c44c\") "
Oct 30 22:33:58.083817 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:33:58.083694    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3ca1201-6968-4e3f-8eb0-4c667cc7c44c-kube-api-access-tkxt8\" (UniqueName: \"kubernetes.io/projected/c3ca1201-6968-4e3f-8eb0-4c667cc7c44c-kube-api-access-tkxt8\") pod \"c3ca1201-6968-4e3f-8eb0-4c667cc7c44c\" (UID: \"c3ca1201-6968-4e3f-8eb0-4c667cc7c44c\") "
Oct 30 22:33:58.083868 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:33:58.083821    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c3ca1201-6968-4e3f-8eb0-4c667cc7c44c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3ca1201-6968-4e3f-8eb0-4c667cc7c44c-maas-api-xks-ca\") pod \"c3ca1201-6968-4e3f-8eb0-4c667cc7c44c\" (UID: \"c3ca1201-6968-4e3f-8eb0-4c667cc7c44c\") "
Oct 30 22:33:58.084098 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:33:58.084074    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3ca1201-6968-4e3f-8eb0-4c667cc7c44c-maas-api-openshift-ca" pod "c3ca1201-6968-4e3f-8eb0-4c667cc7c44c" (UID: "c3ca1201-6968-4e3f-8eb0-4c667cc7c44c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:33:58.084209 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:33:58.084189    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3ca1201-6968-4e3f-8eb0-4c667cc7c44c-maas-api-openshift-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:33:58.085916 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:33:58.085887    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3ca1201-6968-4e3f-8eb0-4c667cc7c44c-kube-api-access-tkxt8" pod "c3ca1201-6968-4e3f-8eb0-4c667cc7c44c" (UID: "c3ca1201-6968-4e3f-8eb0-4c667cc7c44c"). InnerVolumeSpecName "kube-api-access-tkxt8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:33:58.085916 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:33:58.085897    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3ca1201-6968-4e3f-8eb0-4c667cc7c44c-maas-api-xks-ca" pod "c3ca1201-6968-4e3f-8eb0-4c667cc7c44c" (UID: "c3ca1201-6968-4e3f-8eb0-4c667cc7c44c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:33:58.092317 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:33:58.092295    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89af4e0a-blf2f"]
Oct 30 22:33:58.185438 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:33:58.185353    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkxt8\" (UniqueName: \"kubernetes.io/projected/c3ca1201-6968-4e3f-8eb0-4c667cc7c44c-kube-api-access-tkxt8\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:33:58.185438 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:33:58.185385    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3ca1201-6968-4e3f-8eb0-4c667cc7c44c-maas-api-xks-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:33:58.527556 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:33:58.527450    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="577603c3-9fdf-42c5-be4f-ac2533a9c4f0" path="/var/lib/kubelet/pods/577603c3-9fdf-42c5-be4f-ac2533a9c4f0/volumes"
Oct 30 22:33:58.893967 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:33:58.893937    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-nvm5h"
Oct 30 22:33:58.893967 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:33:58.893952    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-nvm5h" event={"ID":"c3ca1201-6968-4e3f-8eb0-4c667cc7c44c","Type":"ContainerDied","Data":"a6d004ce01ab310409942c8d6e9538c614d6a9c8ba7d88e01f58a4a2ecc58c73"}
Oct 30 22:33:58.894185 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:33:58.893998    5061 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a6d004ce01ab310409942c8d6e9538c614d6a9c8ba7d88e01f58a4a2ecc58c73"
Oct 30 22:34:06.858060 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:34:06.858025    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8t2sw"]
Oct 30 22:34:06.858738 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:34:06.858710    5061 cpu_manager.go:414] "containerMap: removing container" podUID="c3ca1201-6968-4e3f-8eb0-4c667cc7c44c" containerName="revoke-keys"
Oct 30 22:34:06.858738 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:34:06.858734    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3ca1201-6968-4e3f-8eb0-4c667cc7c44c" containerName="revoke-keys"
Oct 30 22:34:06.858913 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:34:06.858825    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3ca1201-6968-4e3f-8eb0-4c667cc7c44c" containerName="revoke-keys"
Oct 30 22:34:06.862139 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:34:06.862122    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8t2sw"
Oct 30 22:34:06.961570 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:34:06.961520    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8t2sw"]
Oct 30 22:34:06.978955 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:34:06.978922    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d079dbb3-5d17-4dc9-9cf2-f07629b9f1f2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-8t2sw\" (UID: \"d079dbb3-5d17-4dc9-9cf2-f07629b9f1f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8t2sw"
Oct 30 22:34:06.979144 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:34:06.978979    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d079dbb3-5d17-4dc9-9cf2-f07629b9f1f2-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-8t2sw\" (UID: \"d079dbb3-5d17-4dc9-9cf2-f07629b9f1f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8t2sw"
Oct 30 22:34:06.979144 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:34:06.979040    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d079dbb3-5d17-4dc9-9cf2-f07629b9f1f2-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-8t2sw\" (UID: \"d079dbb3-5d17-4dc9-9cf2-f07629b9f1f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8t2sw"
Oct 30 22:34:06.979144 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:34:06.979098    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d079dbb3-5d17-4dc9-9cf2-f07629b9f1f2-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-8t2sw\" (UID: \"d079dbb3-5d17-4dc9-9cf2-f07629b9f1f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8t2sw"
Oct 30 22:34:06.979144 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:34:06.979130    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7vfv\" (UniqueName: \"kubernetes.io/projected/d079dbb3-5d17-4dc9-9cf2-f07629b9f1f2-kube-api-access-b7vfv\") pod \"maas-default-gateway-openshift-default-694db55c87-8t2sw\" (UID: \"d079dbb3-5d17-4dc9-9cf2-f07629b9f1f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8t2sw"
Oct 30 22:34:06.979311 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:34:06.979161    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d079dbb3-5d17-4dc9-9cf2-f07629b9f1f2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-8t2sw\" (UID: \"d079dbb3-5d17-4dc9-9cf2-f07629b9f1f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8t2sw"
Oct 30 22:34:06.979311 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:34:06.979271    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d079dbb3-5d17-4dc9-9cf2-f07629b9f1f2-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-8t2sw\" (UID: \"d079dbb3-5d17-4dc9-9cf2-f07629b9f1f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8t2sw"
Oct 30 22:34:06.979413 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:34:06.979311    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d079dbb3-5d17-4dc9-9cf2-f07629b9f1f2-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-8t2sw\" (UID: \"d079dbb3-5d17-4dc9-9cf2-f07629b9f1f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8t2sw"
Oct 30 22:34:06.979474 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:34:06.979416    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d079dbb3-5d17-4dc9-9cf2-f07629b9f1f2-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-8t2sw\" (UID: \"d079dbb3-5d17-4dc9-9cf2-f07629b9f1f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8t2sw"
Oct 30 22:34:07.080355 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:34:07.080318    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d079dbb3-5d17-4dc9-9cf2-f07629b9f1f2-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-8t2sw\" (UID: \"d079dbb3-5d17-4dc9-9cf2-f07629b9f1f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8t2sw"
Oct 30 22:34:07.080608 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:34:07.080363    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d079dbb3-5d17-4dc9-9cf2-f07629b9f1f2-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-8t2sw\" (UID: \"d079dbb3-5d17-4dc9-9cf2-f07629b9f1f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8t2sw"
Oct 30 22:34:07.080608 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:34:07.080388    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d079dbb3-5d17-4dc9-9cf2-f07629b9f1f2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-8t2sw\" (UID: \"d079dbb3-5d17-4dc9-9cf2-f07629b9f1f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8t2sw"
Oct 30 22:34:07.080608 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:34:07.080421    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d079dbb3-5d17-4dc9-9cf2-f07629b9f1f2-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-8t2sw\" (UID: \"d079dbb3-5d17-4dc9-9cf2-f07629b9f1f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8t2sw"
Oct 30 22:34:07.080608 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:34:07.080440    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d079dbb3-5d17-4dc9-9cf2-f07629b9f1f2-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-8t2sw\" (UID: \"d079dbb3-5d17-4dc9-9cf2-f07629b9f1f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8t2sw"
Oct 30 22:34:07.080608 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:34:07.080464    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d079dbb3-5d17-4dc9-9cf2-f07629b9f1f2-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-8t2sw\" (UID: \"d079dbb3-5d17-4dc9-9cf2-f07629b9f1f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8t2sw"
Oct 30 22:34:07.080608 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:34:07.080481    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7vfv\" (UniqueName: \"kubernetes.io/projected/d079dbb3-5d17-4dc9-9cf2-f07629b9f1f2-kube-api-access-b7vfv\") pod \"maas-default-gateway-openshift-default-694db55c87-8t2sw\" (UID: \"d079dbb3-5d17-4dc9-9cf2-f07629b9f1f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8t2sw"
Oct 30 22:34:07.080608 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:34:07.080517    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d079dbb3-5d17-4dc9-9cf2-f07629b9f1f2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-8t2sw\" (UID: \"d079dbb3-5d17-4dc9-9cf2-f07629b9f1f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8t2sw"
Oct 30 22:34:07.080957 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:34:07.080623    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d079dbb3-5d17-4dc9-9cf2-f07629b9f1f2-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-8t2sw\" (UID: \"d079dbb3-5d17-4dc9-9cf2-f07629b9f1f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8t2sw"
Oct 30 22:34:07.081030 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:34:07.080951    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d079dbb3-5d17-4dc9-9cf2-f07629b9f1f2-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-8t2sw\" (UID: \"d079dbb3-5d17-4dc9-9cf2-f07629b9f1f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8t2sw"
Oct 30 22:34:07.081030 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:34:07.080977    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d079dbb3-5d17-4dc9-9cf2-f07629b9f1f2-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-8t2sw\" (UID: \"d079dbb3-5d17-4dc9-9cf2-f07629b9f1f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8t2sw"
Oct 30 22:34:07.081148 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:34:07.081045    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d079dbb3-5d17-4dc9-9cf2-f07629b9f1f2-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-8t2sw\" (UID: \"d079dbb3-5d17-4dc9-9cf2-f07629b9f1f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8t2sw"
Oct 30 22:34:07.081248 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:34:07.081229    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d079dbb3-5d17-4dc9-9cf2-f07629b9f1f2-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-8t2sw\" (UID: \"d079dbb3-5d17-4dc9-9cf2-f07629b9f1f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8t2sw"
Oct 30 22:34:07.081569 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:34:07.081523    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d079dbb3-5d17-4dc9-9cf2-f07629b9f1f2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-8t2sw\" (UID: \"d079dbb3-5d17-4dc9-9cf2-f07629b9f1f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8t2sw"
Oct 30 22:34:07.083046 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:34:07.083024    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d079dbb3-5d17-4dc9-9cf2-f07629b9f1f2-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-8t2sw\" (UID: \"d079dbb3-5d17-4dc9-9cf2-f07629b9f1f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8t2sw"
Oct 30 22:34:07.083321 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:34:07.083301    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d079dbb3-5d17-4dc9-9cf2-f07629b9f1f2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-8t2sw\" (UID: \"d079dbb3-5d17-4dc9-9cf2-f07629b9f1f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8t2sw"
Oct 30 22:34:07.135248 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:34:07.135155    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d079dbb3-5d17-4dc9-9cf2-f07629b9f1f2-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-8t2sw\" (UID: \"d079dbb3-5d17-4dc9-9cf2-f07629b9f1f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8t2sw"
Oct 30 22:34:07.135427 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:34:07.135287    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7vfv\" (UniqueName: \"kubernetes.io/projected/d079dbb3-5d17-4dc9-9cf2-f07629b9f1f2-kube-api-access-b7vfv\") pod \"maas-default-gateway-openshift-default-694db55c87-8t2sw\" (UID: \"d079dbb3-5d17-4dc9-9cf2-f07629b9f1f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8t2sw"
Oct 30 22:34:07.173942 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:34:07.173904    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8t2sw"
Oct 30 22:34:07.592326 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:34:07.592296    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8t2sw"]
Oct 30 22:34:07.592451 ip-10-0-130-60 kubenswrapper[5061]: W1030 22:34:07.592348    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd079dbb3_5d17_4dc9_9cf2_f07629b9f1f2.slice/crio-17af31214bcf863eaab658c1f25b04a151bfbe717e1f8d709fe70bb9f6966bd6 WatchSource:0}: Error finding container 17af31214bcf863eaab658c1f25b04a151bfbe717e1f8d709fe70bb9f6966bd6: Status 404 returned error can't find the container with id 17af31214bcf863eaab658c1f25b04a151bfbe717e1f8d709fe70bb9f6966bd6
Oct 30 22:34:07.594901 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:34:07.594870    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:34:07.594991 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:34:07.594947    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:34:07.595040 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:34:07.594993    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:34:07.945297 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:34:07.945208    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8t2sw" event={"ID":"d079dbb3-5d17-4dc9-9cf2-f07629b9f1f2","Type":"ContainerStarted","Data":"1743429f8231d2cfe18aa455fd790757d0745debde6ce79849bd77409335c9fc"}
Oct 30 22:34:07.945297 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:34:07.945261    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8t2sw" event={"ID":"d079dbb3-5d17-4dc9-9cf2-f07629b9f1f2","Type":"ContainerStarted","Data":"17af31214bcf863eaab658c1f25b04a151bfbe717e1f8d709fe70bb9f6966bd6"}
Oct 30 22:34:08.059181 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:34:08.059126    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8t2sw" podStartSLOduration=2.059112575 podStartE2EDuration="2.059112575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:34:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:34:08.057223969 +0000 UTC m=+3101.669993753" watchObservedRunningTime="2026-10-30 22:34:08.059112575 +0000 UTC m=+3101.671882323"
Oct 30 22:34:08.174852 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:34:08.174774    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8t2sw"
Oct 30 22:34:08.176261 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:34:08.176233    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-8t2sw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body=
Oct 30 22:34:08.176358 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:34:08.176281    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8t2sw" podUID="d079dbb3-5d17-4dc9-9cf2-f07629b9f1f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused"
Oct 30 22:34:09.174927 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:34:09.174891    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-8t2sw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body=
Oct 30 22:34:09.175427 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:34:09.174951    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8t2sw" podUID="d079dbb3-5d17-4dc9-9cf2-f07629b9f1f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused"
Oct 30 22:34:10.174914 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:34:10.174860    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-8t2sw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body=
Oct 30 22:34:10.175139 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:34:10.174927    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8t2sw" podUID="d079dbb3-5d17-4dc9-9cf2-f07629b9f1f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused"
Oct 30 22:34:11.175244 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:34:11.175204    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-8t2sw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body=
Oct 30 22:34:11.175782 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:34:11.175266    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8t2sw" podUID="d079dbb3-5d17-4dc9-9cf2-f07629b9f1f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused"
Oct 30 22:34:12.174241 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:34:12.174205    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-8t2sw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body=
Oct 30 22:34:12.174480 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:34:12.174278    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8t2sw" podUID="d079dbb3-5d17-4dc9-9cf2-f07629b9f1f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused"
Oct 30 22:34:13.175031 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:34:13.174986    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-8t2sw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body=
Oct 30 22:34:13.175564 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:34:13.175064    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8t2sw" podUID="d079dbb3-5d17-4dc9-9cf2-f07629b9f1f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused"
Oct 30 22:34:14.176971 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:34:14.176933    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-8t2sw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body=
Oct 30 22:34:14.177643 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:34:14.177590    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8t2sw" podUID="d079dbb3-5d17-4dc9-9cf2-f07629b9f1f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused"
Oct 30 22:34:15.181557 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:34:15.177901    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-8t2sw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body=
Oct 30 22:34:15.181557 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:34:15.177979    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8t2sw" podUID="d079dbb3-5d17-4dc9-9cf2-f07629b9f1f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused"
Oct 30 22:34:17.181139 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:34:17.181088    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-8t2sw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:34:17.181706 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:34:17.181173    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8t2sw" podUID="d079dbb3-5d17-4dc9-9cf2-f07629b9f1f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:34:18.222750 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:34:18.222706    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-8t2sw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:34:18.223286 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:34:18.222772    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8t2sw" podUID="d079dbb3-5d17-4dc9-9cf2-f07629b9f1f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:34:19.223739 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:34:19.223691    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-8t2sw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:34:19.224181 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:34:19.223767    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8t2sw" podUID="d079dbb3-5d17-4dc9-9cf2-f07629b9f1f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:34:20.265741 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:34:20.265695    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-8t2sw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:34:20.266352 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:34:20.265772    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8t2sw" podUID="d079dbb3-5d17-4dc9-9cf2-f07629b9f1f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:34:20.434605 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:34:20.434575    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8t2sw"
Oct 30 22:34:20.435012 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:34:20.434984    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8t2sw"
Oct 30 22:34:20.435706 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:34:20.435687    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8t2sw"
Oct 30 22:34:24.666003 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:34:24.665958    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 22:34:29.510377 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:34:29.510341    5061 scope.go:122] "RemoveContainer" containerID="134457dc3ce8eba0244b97d9913e2c2ac75d58ca030280fdc9e4da116bd3c3ea"
Oct 30 22:34:29.521402 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:34:29.521381    5061 scope.go:122] "RemoveContainer" containerID="d438dcc9f5fa071c157231d56ac38fc7735606d45fe5837862512a2178bc2b80"
Oct 30 22:34:54.666156 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:34:54.666118    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 22:35:24.666702 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:35:24.666662    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 22:35:29.550549 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:35:29.549898    5061 scope.go:122] "RemoveContainer" containerID="ba65bc0bb1e8841ec2f2c0b1550de5ae975998183e7a5a6ec4028aae2167ada1"
Oct 30 22:35:29.603240 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:35:29.603207    5061 scope.go:122] "RemoveContainer" containerID="c930146fb09d07e726e7cfc57e0c53222dcb6761002bc6440af5104f6cfab684"
Oct 30 22:35:29.684560 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:35:29.677686    5061 scope.go:122] "RemoveContainer" containerID="8020972ab398af65f33a70a827749f560ddbb4e2fada89888a5cd3543ecd0a65"
Oct 30 22:35:54.666456 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:35:54.666361    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 22:36:24.666185 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:36:24.666143    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 22:36:29.778591 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:36:29.778554    5061 scope.go:122] "RemoveContainer" containerID="ab1ec9cad2b94c34faf4425e9b84ee08907baf0ccdb28aa5c0e5d207396fd2ca"
Oct 30 22:36:46.053543 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:36:46.053494    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-718807d0-hl455"]
Oct 30 22:36:46.106017 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:36:46.105983    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-718807d0-hl455"]
Oct 30 22:36:46.527194 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:36:46.527120    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2caef524-f1b2-4356-a253-66a13d80d8ca" path="/var/lib/kubelet/pods/2caef524-f1b2-4356-a253-66a13d80d8ca/volumes"
Oct 30 22:36:54.666032 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:36:54.665989    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 22:37:24.667426 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:37:24.667388    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 22:37:26.995593 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:37:26.995564    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-sclrt_5e74071e-0462-42b5-a05c-d90bb638478a/console-operator/2.log"
Oct 30 22:37:27.037051 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:37:27.037023    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-sclrt_5e74071e-0462-42b5-a05c-d90bb638478a/console-operator/2.log"
Oct 30 22:37:29.799926 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:37:29.799891    5061 scope.go:122] "RemoveContainer" containerID="ef8f004856ad46c8dd2bb240b23a4f4717154145c492819c66d8ebf245641cc7"
Oct 30 22:37:29.809050 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:37:29.809017    5061 scope.go:122] "RemoveContainer" containerID="2affb77087b92c8c42090ae65d4419ef2ab4e1c187fcd8c22244217cb1bf9fd5"
Oct 30 22:37:29.816634 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:37:29.816616    5061 scope.go:122] "RemoveContainer" containerID="0ea7e6cd635be4b5d9960f6989a6f918b5fa5dafad74f9b99255efaf414aa511"
Oct 30 22:37:41.546694 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:37:41.546643    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-nbmc8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:37:41.547202 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:37:41.547171    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-nbmc8" podUID="a77dd560-9aff-4292-b430-fa9c72d8c053" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:37:50.049664 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:37:50.049628    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-6hts8"]
Oct 30 22:37:50.103354 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:37:50.103326    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-6hts8"]
Oct 30 22:37:50.527699 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:37:50.527621    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5eb3cbe-605a-44a9-b51e-a2da5ab313b8" path="/var/lib/kubelet/pods/f5eb3cbe-605a-44a9-b51e-a2da5ab313b8/volumes"
Oct 30 22:37:54.666449 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:37:54.666413    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 22:38:24.666906 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:38:24.666868    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 22:38:29.843540 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:38:29.843487    5061 scope.go:122] "RemoveContainer" containerID="890a6158fc31b9e035338bcb1db222e09ce77c077ea8fea10b695ab0894ddb4b"
Oct 30 22:38:54.667195 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:38:54.667111    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 22:38:59.051736 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:38:59.051699    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-nvm5h"]
Oct 30 22:38:59.104029 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:38:59.103994    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-nvm5h"]
Oct 30 22:39:00.527793 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:39:00.527757    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3ca1201-6968-4e3f-8eb0-4c667cc7c44c" path="/var/lib/kubelet/pods/c3ca1201-6968-4e3f-8eb0-4c667cc7c44c/volumes"
Oct 30 22:39:24.667338 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:39:24.667300    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 22:39:29.864669 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:39:29.864637    5061 scope.go:122] "RemoveContainer" containerID="df54277ebcb9e9926995483481e503296b08a16e04990e14b3366138273f41ee"
Oct 30 22:39:46.302553 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:39:46.301749    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-8pzt2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:39:46.302553 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:39:46.301845    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8pzt2" podUID="b3808649-2258-4508-be89-8e0c57ca83e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:39:54.804256 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:39:54.804212    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 22:40:24.666895 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:40:24.666852    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 22:40:29.888162 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:40:29.888128    5061 scope.go:122] "RemoveContainer" containerID="66bf3e4b823385b1ec2ae8ca6c5c9e434aa2d67eaf59525a3f3eb545260e2b48"
Oct 30 22:40:33.928416 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:40:33.928378    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-673c84-d468879f85skhz"]
Oct 30 22:40:33.932148 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:40:33.932130    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-673c84-d468879f85skhz"
Oct 30 22:40:33.989520 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:40:33.989490    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-673c84-d468879f85skhz"]
Oct 30 22:40:34.018185 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:40:34.018156    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76765e23-d419-43c7-a763-18224674780d-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-673c84-d468879f85skhz\" (UID: \"76765e23-d419-43c7-a763-18224674780d\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-673c84-d468879f85skhz"
Oct 30 22:40:34.018327 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:40:34.018209    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkjj9\" (UniqueName: \"kubernetes.io/projected/76765e23-d419-43c7-a763-18224674780d-kube-api-access-lkjj9\") pod \"payload-processing-e2e-worker-single-main-673c84-d468879f85skhz\" (UID: \"76765e23-d419-43c7-a763-18224674780d\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-673c84-d468879f85skhz"
Oct 30 22:40:34.040075 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:40:34.040047    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-4745d3dc-dockercfg-24xzm\""
Oct 30 22:40:34.040179 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:40:34.040096    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-673c84\""
Oct 30 22:40:34.067839 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:40:34.067814    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-673c84-d7d4fgxr4m"]
Oct 30 22:40:34.071424 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:40:34.071408    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-673c84-d7d4fgxr4m"
Oct 30 22:40:34.119386 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:40:34.119357    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ae521ec-c315-47a0-a700-d98dc9512a47-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-673c84-d7d4fgxr4m\" (UID: \"0ae521ec-c315-47a0-a700-d98dc9512a47\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-673c84-d7d4fgxr4m"
Oct 30 22:40:34.119501 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:40:34.119475    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76765e23-d419-43c7-a763-18224674780d-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-673c84-d468879f85skhz\" (UID: \"76765e23-d419-43c7-a763-18224674780d\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-673c84-d468879f85skhz"
Oct 30 22:40:34.119591 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:40:34.119513    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsjrh\" (UniqueName: \"kubernetes.io/projected/0ae521ec-c315-47a0-a700-d98dc9512a47-kube-api-access-zsjrh\") pod \"payload-pre-processing-e2e-worker-single-main-673c84-d7d4fgxr4m\" (UID: \"0ae521ec-c315-47a0-a700-d98dc9512a47\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-673c84-d7d4fgxr4m"
Oct 30 22:40:34.119591 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:40:34.119572    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkjj9\" (UniqueName: \"kubernetes.io/projected/76765e23-d419-43c7-a763-18224674780d-kube-api-access-lkjj9\") pod \"payload-processing-e2e-worker-single-main-673c84-d468879f85skhz\" (UID: \"76765e23-d419-43c7-a763-18224674780d\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-673c84-d468879f85skhz"
Oct 30 22:40:34.120217 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:40:34.120200    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76765e23-d419-43c7-a763-18224674780d-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-673c84-d468879f85skhz\" (UID: \"76765e23-d419-43c7-a763-18224674780d\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-673c84-d468879f85skhz"
Oct 30 22:40:34.130418 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:40:34.130394    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-673c84-d7d4fgxr4m"]
Oct 30 22:40:34.179676 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:40:34.179607    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkjj9\" (UniqueName: \"kubernetes.io/projected/76765e23-d419-43c7-a763-18224674780d-kube-api-access-lkjj9\") pod \"payload-processing-e2e-worker-single-main-673c84-d468879f85skhz\" (UID: \"76765e23-d419-43c7-a763-18224674780d\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-673c84-d468879f85skhz"
Oct 30 22:40:34.220390 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:40:34.220352    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ae521ec-c315-47a0-a700-d98dc9512a47-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-673c84-d7d4fgxr4m\" (UID: \"0ae521ec-c315-47a0-a700-d98dc9512a47\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-673c84-d7d4fgxr4m"
Oct 30 22:40:34.220568 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:40:34.220437    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsjrh\" (UniqueName: \"kubernetes.io/projected/0ae521ec-c315-47a0-a700-d98dc9512a47-kube-api-access-zsjrh\") pod \"payload-pre-processing-e2e-worker-single-main-673c84-d7d4fgxr4m\" (UID: \"0ae521ec-c315-47a0-a700-d98dc9512a47\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-673c84-d7d4fgxr4m"
Oct 30 22:40:34.221136 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:40:34.221113    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ae521ec-c315-47a0-a700-d98dc9512a47-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-673c84-d7d4fgxr4m\" (UID: \"0ae521ec-c315-47a0-a700-d98dc9512a47\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-673c84-d7d4fgxr4m"
Oct 30 22:40:34.245281 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:40:34.245256    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-673c84-d468879f85skhz"
Oct 30 22:40:34.280638 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:40:34.280602    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsjrh\" (UniqueName: \"kubernetes.io/projected/0ae521ec-c315-47a0-a700-d98dc9512a47-kube-api-access-zsjrh\") pod \"payload-pre-processing-e2e-worker-single-main-673c84-d7d4fgxr4m\" (UID: \"0ae521ec-c315-47a0-a700-d98dc9512a47\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-673c84-d7d4fgxr4m"
Oct 30 22:40:34.387304 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:40:34.387257    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-673c84-d7d4fgxr4m"
Oct 30 22:40:34.610690 ip-10-0-130-60 kubenswrapper[5061]: W1030 22:40:34.610644    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76765e23_d419_43c7_a763_18224674780d.slice/crio-dbea6f683247a78319f6386add1d929e5417796be05595bd13b9ace2b9537ff7 WatchSource:0}: Error finding container dbea6f683247a78319f6386add1d929e5417796be05595bd13b9ace2b9537ff7: Status 404 returned error can't find the container with id dbea6f683247a78319f6386add1d929e5417796be05595bd13b9ace2b9537ff7
Oct 30 22:40:34.615863 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:40:34.615839    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-673c84-d468879f85skhz"]
Oct 30 22:40:34.752405 ip-10-0-130-60 kubenswrapper[5061]: W1030 22:40:34.752377    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ae521ec_c315_47a0_a700_d98dc9512a47.slice/crio-3b46f330f407caeb336888a25ff774b1a05da3199723069b10d8ce302d0a9fcd WatchSource:0}: Error finding container 3b46f330f407caeb336888a25ff774b1a05da3199723069b10d8ce302d0a9fcd: Status 404 returned error can't find the container with id 3b46f330f407caeb336888a25ff774b1a05da3199723069b10d8ce302d0a9fcd
Oct 30 22:40:34.756954 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:40:34.756925    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-673c84-d7d4fgxr4m"]
Oct 30 22:40:35.399410 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:40:35.399371    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-673c84-d468879f85skhz" event={"ID":"76765e23-d419-43c7-a763-18224674780d","Type":"ContainerStarted","Data":"ebb91ea58a0363984f411d3a22b8c6c9a5f1a19a8294e852a819cb47885e49f3"}
Oct 30 22:40:35.399905 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:40:35.399552    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-673c84-d468879f85skhz"
Oct 30 22:40:35.399905 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:40:35.399581    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-673c84-d468879f85skhz" event={"ID":"76765e23-d419-43c7-a763-18224674780d","Type":"ContainerStarted","Data":"dbea6f683247a78319f6386add1d929e5417796be05595bd13b9ace2b9537ff7"}
Oct 30 22:40:35.400970 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:40:35.400936    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-673c84-d7d4fgxr4m" event={"ID":"0ae521ec-c315-47a0-a700-d98dc9512a47","Type":"ContainerStarted","Data":"d9823370606408df9f2ec11b0a79f3e2264f43d0b03515f3cb8425b3e262203a"}
Oct 30 22:40:35.401095 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:40:35.400981    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-673c84-d7d4fgxr4m"
Oct 30 22:40:35.401095 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:40:35.400997    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-673c84-d7d4fgxr4m" event={"ID":"0ae521ec-c315-47a0-a700-d98dc9512a47","Type":"ContainerStarted","Data":"3b46f330f407caeb336888a25ff774b1a05da3199723069b10d8ce302d0a9fcd"}
Oct 30 22:40:35.521235 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:40:35.521190    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-673c84-d468879f85skhz" podStartSLOduration=2.521176698 podStartE2EDuration="2.521176698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:40:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:40:35.519246716 +0000 UTC m=+3489.132016475" watchObservedRunningTime="2026-10-30 22:40:35.521176698 +0000 UTC m=+3489.133946456"
Oct 30 22:40:35.640982 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:40:35.640905    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-673c84-d7d4fgxr4m" podStartSLOduration=1.640887293 podStartE2EDuration="1.640887293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:40:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:40:35.640251524 +0000 UTC m=+3489.253021308" watchObservedRunningTime="2026-10-30 22:40:35.640887293 +0000 UTC m=+3489.253657052"
Oct 30 22:40:46.409508 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:40:46.409475    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-673c84-d468879f85skhz"
Oct 30 22:40:46.409904 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:40:46.409682    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-673c84-d7d4fgxr4m"
Oct 30 22:40:54.666816 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:40:54.666777    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 22:41:19.454321 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:41:19.454273    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-673c84/simulator-single-main-673c84-kserve-85c7949ff-74bxf"]
Oct 30 22:41:19.458047 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:41:19.458030    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-673c84/simulator-single-main-673c84-kserve-85c7949ff-74bxf"
Oct 30 22:41:19.514359 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:41:19.514321    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-673c84\"/\"openshift-service-ca.crt\""
Oct 30 22:41:19.566820 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:41:19.566791    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-673c84\"/\"kube-root-ca.crt\""
Oct 30 22:41:19.567012 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:41:19.566997    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-673c84\"/\"default-dockercfg-8gt5w\""
Oct 30 22:41:19.567080 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:41:19.567004    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-673c84\"/\"simulator-single-main-673c84-kserve-self-signed-certs\""
Oct 30 22:41:19.569158 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:41:19.569138    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-673c84/simulator-single-main-673c84-kserve-85c7949ff-74bxf"]
Oct 30 22:41:19.579733 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:41:19.579708    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19a817cf-0058-4865-8517-415419b98a26-tmp-dir\") pod \"simulator-single-main-673c84-kserve-85c7949ff-74bxf\" (UID: \"19a817cf-0058-4865-8517-415419b98a26\") " pod="e2e-models-e2e-worker-single-main-673c84/simulator-single-main-673c84-kserve-85c7949ff-74bxf"
Oct 30 22:41:19.579884 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:41:19.579754    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19a817cf-0058-4865-8517-415419b98a26-model-cache\") pod \"simulator-single-main-673c84-kserve-85c7949ff-74bxf\" (UID: \"19a817cf-0058-4865-8517-415419b98a26\") " pod="e2e-models-e2e-worker-single-main-673c84/simulator-single-main-673c84-kserve-85c7949ff-74bxf"
Oct 30 22:41:19.579884 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:41:19.579836    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19a817cf-0058-4865-8517-415419b98a26-dshm\") pod \"simulator-single-main-673c84-kserve-85c7949ff-74bxf\" (UID: \"19a817cf-0058-4865-8517-415419b98a26\") " pod="e2e-models-e2e-worker-single-main-673c84/simulator-single-main-673c84-kserve-85c7949ff-74bxf"
Oct 30 22:41:19.580014 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:41:19.579886    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19a817cf-0058-4865-8517-415419b98a26-home\") pod \"simulator-single-main-673c84-kserve-85c7949ff-74bxf\" (UID: \"19a817cf-0058-4865-8517-415419b98a26\") " pod="e2e-models-e2e-worker-single-main-673c84/simulator-single-main-673c84-kserve-85c7949ff-74bxf"
Oct 30 22:41:19.580014 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:41:19.579912    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19a817cf-0058-4865-8517-415419b98a26-tls-certs\") pod \"simulator-single-main-673c84-kserve-85c7949ff-74bxf\" (UID: \"19a817cf-0058-4865-8517-415419b98a26\") " pod="e2e-models-e2e-worker-single-main-673c84/simulator-single-main-673c84-kserve-85c7949ff-74bxf"
Oct 30 22:41:19.580014 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:41:19.579949    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6r984\" (UniqueName: \"kubernetes.io/projected/19a817cf-0058-4865-8517-415419b98a26-kube-api-access-6r984\") pod \"simulator-single-main-673c84-kserve-85c7949ff-74bxf\" (UID: \"19a817cf-0058-4865-8517-415419b98a26\") " pod="e2e-models-e2e-worker-single-main-673c84/simulator-single-main-673c84-kserve-85c7949ff-74bxf"
Oct 30 22:41:19.681139 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:41:19.681041    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19a817cf-0058-4865-8517-415419b98a26-tmp-dir\") pod \"simulator-single-main-673c84-kserve-85c7949ff-74bxf\" (UID: \"19a817cf-0058-4865-8517-415419b98a26\") " pod="e2e-models-e2e-worker-single-main-673c84/simulator-single-main-673c84-kserve-85c7949ff-74bxf"
Oct 30 22:41:19.681139 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:41:19.681105    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19a817cf-0058-4865-8517-415419b98a26-model-cache\") pod \"simulator-single-main-673c84-kserve-85c7949ff-74bxf\" (UID: \"19a817cf-0058-4865-8517-415419b98a26\") " pod="e2e-models-e2e-worker-single-main-673c84/simulator-single-main-673c84-kserve-85c7949ff-74bxf"
Oct 30 22:41:19.681139 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:41:19.681132    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19a817cf-0058-4865-8517-415419b98a26-dshm\") pod \"simulator-single-main-673c84-kserve-85c7949ff-74bxf\" (UID: \"19a817cf-0058-4865-8517-415419b98a26\") " pod="e2e-models-e2e-worker-single-main-673c84/simulator-single-main-673c84-kserve-85c7949ff-74bxf"
Oct 30 22:41:19.681433 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:41:19.681173    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19a817cf-0058-4865-8517-415419b98a26-home\") pod \"simulator-single-main-673c84-kserve-85c7949ff-74bxf\" (UID: \"19a817cf-0058-4865-8517-415419b98a26\") " pod="e2e-models-e2e-worker-single-main-673c84/simulator-single-main-673c84-kserve-85c7949ff-74bxf"
Oct 30 22:41:19.681433 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:41:19.681196    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19a817cf-0058-4865-8517-415419b98a26-tls-certs\") pod \"simulator-single-main-673c84-kserve-85c7949ff-74bxf\" (UID: \"19a817cf-0058-4865-8517-415419b98a26\") " pod="e2e-models-e2e-worker-single-main-673c84/simulator-single-main-673c84-kserve-85c7949ff-74bxf"
Oct 30 22:41:19.681433 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:41:19.681221    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6r984\" (UniqueName: \"kubernetes.io/projected/19a817cf-0058-4865-8517-415419b98a26-kube-api-access-6r984\") pod \"simulator-single-main-673c84-kserve-85c7949ff-74bxf\" (UID: \"19a817cf-0058-4865-8517-415419b98a26\") " pod="e2e-models-e2e-worker-single-main-673c84/simulator-single-main-673c84-kserve-85c7949ff-74bxf"
Oct 30 22:41:19.681611 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:41:19.681584    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19a817cf-0058-4865-8517-415419b98a26-tmp-dir\") pod \"simulator-single-main-673c84-kserve-85c7949ff-74bxf\" (UID: \"19a817cf-0058-4865-8517-415419b98a26\") " pod="e2e-models-e2e-worker-single-main-673c84/simulator-single-main-673c84-kserve-85c7949ff-74bxf"
Oct 30 22:41:19.681695 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:41:19.681609    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19a817cf-0058-4865-8517-415419b98a26-home\") pod \"simulator-single-main-673c84-kserve-85c7949ff-74bxf\" (UID: \"19a817cf-0058-4865-8517-415419b98a26\") " pod="e2e-models-e2e-worker-single-main-673c84/simulator-single-main-673c84-kserve-85c7949ff-74bxf"
Oct 30 22:41:19.681695 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:41:19.681649    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19a817cf-0058-4865-8517-415419b98a26-model-cache\") pod \"simulator-single-main-673c84-kserve-85c7949ff-74bxf\" (UID: \"19a817cf-0058-4865-8517-415419b98a26\") " pod="e2e-models-e2e-worker-single-main-673c84/simulator-single-main-673c84-kserve-85c7949ff-74bxf"
Oct 30 22:41:19.683556 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:41:19.683519    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19a817cf-0058-4865-8517-415419b98a26-dshm\") pod \"simulator-single-main-673c84-kserve-85c7949ff-74bxf\" (UID: \"19a817cf-0058-4865-8517-415419b98a26\") " pod="e2e-models-e2e-worker-single-main-673c84/simulator-single-main-673c84-kserve-85c7949ff-74bxf"
Oct 30 22:41:19.683881 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:41:19.683864    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19a817cf-0058-4865-8517-415419b98a26-tls-certs\") pod \"simulator-single-main-673c84-kserve-85c7949ff-74bxf\" (UID: \"19a817cf-0058-4865-8517-415419b98a26\") " pod="e2e-models-e2e-worker-single-main-673c84/simulator-single-main-673c84-kserve-85c7949ff-74bxf"
Oct 30 22:41:19.742368 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:41:19.742337    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6r984\" (UniqueName: \"kubernetes.io/projected/19a817cf-0058-4865-8517-415419b98a26-kube-api-access-6r984\") pod \"simulator-single-main-673c84-kserve-85c7949ff-74bxf\" (UID: \"19a817cf-0058-4865-8517-415419b98a26\") " pod="e2e-models-e2e-worker-single-main-673c84/simulator-single-main-673c84-kserve-85c7949ff-74bxf"
Oct 30 22:41:19.770156 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:41:19.770124    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-673c84/simulator-single-main-673c84-kserve-85c7949ff-74bxf"
Oct 30 22:41:20.141516 ip-10-0-130-60 kubenswrapper[5061]: W1030 22:41:20.141468    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod19a817cf_0058_4865_8517_415419b98a26.slice/crio-c1b039c48291f88d0ef0eebe7a921f2c6a9fb974c0ce2e5fe4d6fbab98c45e52 WatchSource:0}: Error finding container c1b039c48291f88d0ef0eebe7a921f2c6a9fb974c0ce2e5fe4d6fbab98c45e52: Status 404 returned error can't find the container with id c1b039c48291f88d0ef0eebe7a921f2c6a9fb974c0ce2e5fe4d6fbab98c45e52
Oct 30 22:41:20.144920 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:41:20.144892    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-673c84/simulator-single-main-673c84-kserve-85c7949ff-74bxf"]
Oct 30 22:41:20.643480 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:41:20.643437    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-673c84/simulator-single-main-673c84-kserve-85c7949ff-74bxf" event={"ID":"19a817cf-0058-4865-8517-415419b98a26","Type":"ContainerStarted","Data":"f17ccd799489818573edd08e085b0da3a0f29f0ef6e8fde2d4255a50be0d7271"}
Oct 30 22:41:20.643884 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:41:20.643495    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-673c84/simulator-single-main-673c84-kserve-85c7949ff-74bxf" event={"ID":"19a817cf-0058-4865-8517-415419b98a26","Type":"ContainerStarted","Data":"c1b039c48291f88d0ef0eebe7a921f2c6a9fb974c0ce2e5fe4d6fbab98c45e52"}
Oct 30 22:41:20.766150 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:41:20.766093    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-673c84/simulator-single-main-673c84-kserve-85c7949ff-74bxf" podStartSLOduration=1.7660797289999999 podStartE2EDuration="1.766079729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:41:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:41:20.763835733 +0000 UTC m=+3534.376605492" watchObservedRunningTime="2026-10-30 22:41:20.766079729 +0000 UTC m=+3534.378849486"
Oct 30 22:41:20.770939 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:41:20.770908    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-673c84/simulator-single-main-673c84-kserve-85c7949ff-74bxf"
Oct 30 22:41:24.668815 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:41:24.668778    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 22:41:29.770808 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:41:29.770774    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-673c84/simulator-single-main-673c84-kserve-85c7949ff-74bxf"
Oct 30 22:41:29.787743 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:41:29.787712    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-673c84/simulator-single-main-673c84-kserve-85c7949ff-74bxf"
Oct 30 22:41:30.707074 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:41:30.707045    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-673c84/simulator-single-main-673c84-kserve-85c7949ff-74bxf"
Oct 30 22:41:54.667521 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:41:54.667480    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 22:42:03.727079 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:03.727036    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac4883ef-5hk5q"]
Oct 30 22:42:03.730721 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:03.730705    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac4883ef-5hk5q"
Oct 30 22:42:03.788826 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:03.788795    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5acee650-6028-4568-b758-fd9f45672ff4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac4883ef-5hk5q\" (UID: \"5acee650-6028-4568-b758-fd9f45672ff4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac4883ef-5hk5q"
Oct 30 22:42:03.788956 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:03.788935    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfmpp\" (UniqueName: \"kubernetes.io/projected/5acee650-6028-4568-b758-fd9f45672ff4-kube-api-access-nfmpp\") pod \"maas-api-revoke-sub-ac4883ef-5hk5q\" (UID: \"5acee650-6028-4568-b758-fd9f45672ff4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac4883ef-5hk5q"
Oct 30 22:42:03.789029 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:03.788985    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5acee650-6028-4568-b758-fd9f45672ff4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac4883ef-5hk5q\" (UID: \"5acee650-6028-4568-b758-fd9f45672ff4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac4883ef-5hk5q"
Oct 30 22:42:03.789132 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:03.789107    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac4883ef-5hk5q"]
Oct 30 22:42:03.838951 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:03.838920    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:42:03.839059 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:03.838954    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:42:03.839059 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:03.838973    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gdmfg\""
Oct 30 22:42:03.839059 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:03.838965    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:42:03.889908 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:03.889883    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5acee650-6028-4568-b758-fd9f45672ff4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac4883ef-5hk5q\" (UID: \"5acee650-6028-4568-b758-fd9f45672ff4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac4883ef-5hk5q"
Oct 30 22:42:03.890022 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:03.889959    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfmpp\" (UniqueName: \"kubernetes.io/projected/5acee650-6028-4568-b758-fd9f45672ff4-kube-api-access-nfmpp\") pod \"maas-api-revoke-sub-ac4883ef-5hk5q\" (UID: \"5acee650-6028-4568-b758-fd9f45672ff4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac4883ef-5hk5q"
Oct 30 22:42:03.890022 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:03.889989    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5acee650-6028-4568-b758-fd9f45672ff4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac4883ef-5hk5q\" (UID: \"5acee650-6028-4568-b758-fd9f45672ff4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac4883ef-5hk5q"
Oct 30 22:42:03.890654 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:03.890633    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5acee650-6028-4568-b758-fd9f45672ff4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac4883ef-5hk5q\" (UID: \"5acee650-6028-4568-b758-fd9f45672ff4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac4883ef-5hk5q"
Oct 30 22:42:03.892086 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:03.892068    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5acee650-6028-4568-b758-fd9f45672ff4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac4883ef-5hk5q\" (UID: \"5acee650-6028-4568-b758-fd9f45672ff4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac4883ef-5hk5q"
Oct 30 22:42:03.950198 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:03.950169    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfmpp\" (UniqueName: \"kubernetes.io/projected/5acee650-6028-4568-b758-fd9f45672ff4-kube-api-access-nfmpp\") pod \"maas-api-revoke-sub-ac4883ef-5hk5q\" (UID: \"5acee650-6028-4568-b758-fd9f45672ff4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac4883ef-5hk5q"
Oct 30 22:42:04.043827 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:04.043754    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac4883ef-5hk5q"
Oct 30 22:42:04.409649 ip-10-0-130-60 kubenswrapper[5061]: W1030 22:42:04.409619    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5acee650_6028_4568_b758_fd9f45672ff4.slice/crio-632cc8ace0292373ccfc5ee6895278ba990c8c9af21d3275b0b50e4b1e2e814a WatchSource:0}: Error finding container 632cc8ace0292373ccfc5ee6895278ba990c8c9af21d3275b0b50e4b1e2e814a: Status 404 returned error can't find the container with id 632cc8ace0292373ccfc5ee6895278ba990c8c9af21d3275b0b50e4b1e2e814a
Oct 30 22:42:04.413135 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:04.413107    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac4883ef-5hk5q"]
Oct 30 22:42:04.872045 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:04.872012    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac4883ef-5hk5q" event={"ID":"5acee650-6028-4568-b758-fd9f45672ff4","Type":"ContainerStarted","Data":"19b70de5518761d3a785e08ad0b74c7dbd032473d6907a5303df296e68e4c342"}
Oct 30 22:42:04.872444 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:04.872069    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac4883ef-5hk5q" event={"ID":"5acee650-6028-4568-b758-fd9f45672ff4","Type":"ContainerStarted","Data":"632cc8ace0292373ccfc5ee6895278ba990c8c9af21d3275b0b50e4b1e2e814a"}
Oct 30 22:42:04.992027 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:04.991932    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac4883ef-5hk5q" podStartSLOduration=1.991918333 podStartE2EDuration="1.991918333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:42:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:42:04.991170013 +0000 UTC m=+3578.603939782" watchObservedRunningTime="2026-10-30 22:42:04.991918333 +0000 UTC m=+3578.604688091"
Oct 30 22:42:05.878567 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:05.878515    5061 generic.go:356] "Generic (PLEG): container finished" podID="5acee650-6028-4568-b758-fd9f45672ff4" containerID="19b70de5518761d3a785e08ad0b74c7dbd032473d6907a5303df296e68e4c342" exitCode=0
Oct 30 22:42:05.878567 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:05.878561    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac4883ef-5hk5q" event={"ID":"5acee650-6028-4568-b758-fd9f45672ff4","Type":"ContainerDied","Data":"19b70de5518761d3a785e08ad0b74c7dbd032473d6907a5303df296e68e4c342"}
Oct 30 22:42:07.030250 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:07.030224    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac4883ef-5hk5q"
Oct 30 22:42:07.126339 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:07.126305    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5acee650-6028-4568-b758-fd9f45672ff4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5acee650-6028-4568-b758-fd9f45672ff4-maas-api-xks-ca\") pod \"5acee650-6028-4568-b758-fd9f45672ff4\" (UID: \"5acee650-6028-4568-b758-fd9f45672ff4\") "
Oct 30 22:42:07.126491 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:07.126400    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5acee650-6028-4568-b758-fd9f45672ff4-kube-api-access-nfmpp\" (UniqueName: \"kubernetes.io/projected/5acee650-6028-4568-b758-fd9f45672ff4-kube-api-access-nfmpp\") pod \"5acee650-6028-4568-b758-fd9f45672ff4\" (UID: \"5acee650-6028-4568-b758-fd9f45672ff4\") "
Oct 30 22:42:07.126491 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:07.126439    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5acee650-6028-4568-b758-fd9f45672ff4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5acee650-6028-4568-b758-fd9f45672ff4-maas-api-openshift-ca\") pod \"5acee650-6028-4568-b758-fd9f45672ff4\" (UID: \"5acee650-6028-4568-b758-fd9f45672ff4\") "
Oct 30 22:42:07.126943 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:07.126907    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5acee650-6028-4568-b758-fd9f45672ff4-maas-api-openshift-ca" pod "5acee650-6028-4568-b758-fd9f45672ff4" (UID: "5acee650-6028-4568-b758-fd9f45672ff4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:42:07.128569 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:07.128539    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5acee650-6028-4568-b758-fd9f45672ff4-kube-api-access-nfmpp" pod "5acee650-6028-4568-b758-fd9f45672ff4" (UID: "5acee650-6028-4568-b758-fd9f45672ff4"). InnerVolumeSpecName "kube-api-access-nfmpp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:42:07.128569 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:07.128516    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5acee650-6028-4568-b758-fd9f45672ff4-maas-api-xks-ca" pod "5acee650-6028-4568-b758-fd9f45672ff4" (UID: "5acee650-6028-4568-b758-fd9f45672ff4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:42:07.228252 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:07.228219    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5acee650-6028-4568-b758-fd9f45672ff4-maas-api-xks-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:42:07.228252 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:07.228246    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfmpp\" (UniqueName: \"kubernetes.io/projected/5acee650-6028-4568-b758-fd9f45672ff4-kube-api-access-nfmpp\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:42:07.228252 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:07.228259    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5acee650-6028-4568-b758-fd9f45672ff4-maas-api-openshift-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:42:07.890230 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:07.890195    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac4883ef-5hk5q" event={"ID":"5acee650-6028-4568-b758-fd9f45672ff4","Type":"ContainerDied","Data":"632cc8ace0292373ccfc5ee6895278ba990c8c9af21d3275b0b50e4b1e2e814a"}
Oct 30 22:42:07.890230 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:07.890221    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac4883ef-5hk5q"
Oct 30 22:42:07.890444 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:07.890245    5061 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="632cc8ace0292373ccfc5ee6895278ba990c8c9af21d3275b0b50e4b1e2e814a"
Oct 30 22:42:15.941465 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:15.941422    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-673c84-frswm"]
Oct 30 22:42:15.942093 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:15.942071    5061 cpu_manager.go:414] "containerMap: removing container" podUID="5acee650-6028-4568-b758-fd9f45672ff4" containerName="revoke-keys"
Oct 30 22:42:15.942179 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:15.942094    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5acee650-6028-4568-b758-fd9f45672ff4" containerName="revoke-keys"
Oct 30 22:42:15.942268 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:15.942254    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="5acee650-6028-4568-b758-fd9f45672ff4" containerName="revoke-keys"
Oct 30 22:42:15.945642 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:15.945621    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-673c84-frswm"
Oct 30 22:42:16.003860 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:16.003823    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-673c84-frswm"]
Oct 30 22:42:16.016390 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:16.016360    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxtrr\" (UniqueName: \"kubernetes.io/projected/41d4fc04-b22d-46a5-85c5-25d46719e3cb-kube-api-access-sxtrr\") pod \"maas-api-revoke-keys-e2e-worker-single-main-673c84-frswm\" (UID: \"41d4fc04-b22d-46a5-85c5-25d46719e3cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-673c84-frswm"
Oct 30 22:42:16.016577 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:16.016397    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41d4fc04-b22d-46a5-85c5-25d46719e3cb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-673c84-frswm\" (UID: \"41d4fc04-b22d-46a5-85c5-25d46719e3cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-673c84-frswm"
Oct 30 22:42:16.016577 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:16.016422    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41d4fc04-b22d-46a5-85c5-25d46719e3cb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-673c84-frswm\" (UID: \"41d4fc04-b22d-46a5-85c5-25d46719e3cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-673c84-frswm"
Oct 30 22:42:16.053574 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:16.053536    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:42:16.053574 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:16.053566    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:42:16.053787 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:16.053636    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gdmfg\""
Oct 30 22:42:16.053787 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:16.053690    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:42:16.117969 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:16.117928    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxtrr\" (UniqueName: \"kubernetes.io/projected/41d4fc04-b22d-46a5-85c5-25d46719e3cb-kube-api-access-sxtrr\") pod \"maas-api-revoke-keys-e2e-worker-single-main-673c84-frswm\" (UID: \"41d4fc04-b22d-46a5-85c5-25d46719e3cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-673c84-frswm"
Oct 30 22:42:16.117969 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:16.117973    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41d4fc04-b22d-46a5-85c5-25d46719e3cb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-673c84-frswm\" (UID: \"41d4fc04-b22d-46a5-85c5-25d46719e3cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-673c84-frswm"
Oct 30 22:42:16.118251 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:16.117995    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41d4fc04-b22d-46a5-85c5-25d46719e3cb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-673c84-frswm\" (UID: \"41d4fc04-b22d-46a5-85c5-25d46719e3cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-673c84-frswm"
Oct 30 22:42:16.118848 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:16.118821    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41d4fc04-b22d-46a5-85c5-25d46719e3cb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-673c84-frswm\" (UID: \"41d4fc04-b22d-46a5-85c5-25d46719e3cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-673c84-frswm"
Oct 30 22:42:16.120368 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:16.120342    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41d4fc04-b22d-46a5-85c5-25d46719e3cb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-673c84-frswm\" (UID: \"41d4fc04-b22d-46a5-85c5-25d46719e3cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-673c84-frswm"
Oct 30 22:42:16.178041 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:16.178004    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxtrr\" (UniqueName: \"kubernetes.io/projected/41d4fc04-b22d-46a5-85c5-25d46719e3cb-kube-api-access-sxtrr\") pod \"maas-api-revoke-keys-e2e-worker-single-main-673c84-frswm\" (UID: \"41d4fc04-b22d-46a5-85c5-25d46719e3cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-673c84-frswm"
Oct 30 22:42:16.258141 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:16.258063    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-673c84-frswm"
Oct 30 22:42:16.631992 ip-10-0-130-60 kubenswrapper[5061]: W1030 22:42:16.631959    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41d4fc04_b22d_46a5_85c5_25d46719e3cb.slice/crio-e67fb9fcbdf8f5a9d4d144be0c3b082b39a778d2275cb8bdd47bc4ede2438cdd WatchSource:0}: Error finding container e67fb9fcbdf8f5a9d4d144be0c3b082b39a778d2275cb8bdd47bc4ede2438cdd: Status 404 returned error can't find the container with id e67fb9fcbdf8f5a9d4d144be0c3b082b39a778d2275cb8bdd47bc4ede2438cdd
Oct 30 22:42:16.635323 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:16.635289    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-673c84-frswm"]
Oct 30 22:42:16.941120 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:16.941082    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-673c84-frswm" event={"ID":"41d4fc04-b22d-46a5-85c5-25d46719e3cb","Type":"ContainerStarted","Data":"e67fb9fcbdf8f5a9d4d144be0c3b082b39a778d2275cb8bdd47bc4ede2438cdd"}
Oct 30 22:42:17.947914 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:17.947808    5061 generic.go:356] "Generic (PLEG): container finished" podID="41d4fc04-b22d-46a5-85c5-25d46719e3cb" containerID="3af2b05941219699ab795d80bacdde95e4e483c670363555929996a556377e09" exitCode=0
Oct 30 22:42:17.947914 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:17.947895    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-673c84-frswm" event={"ID":"41d4fc04-b22d-46a5-85c5-25d46719e3cb","Type":"ContainerDied","Data":"3af2b05941219699ab795d80bacdde95e4e483c670363555929996a556377e09"}
Oct 30 22:42:19.118227 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:19.118195    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-673c84-frswm"
Oct 30 22:42:19.255655 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:19.255570    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41d4fc04-b22d-46a5-85c5-25d46719e3cb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41d4fc04-b22d-46a5-85c5-25d46719e3cb-maas-api-openshift-ca\") pod \"41d4fc04-b22d-46a5-85c5-25d46719e3cb\" (UID: \"41d4fc04-b22d-46a5-85c5-25d46719e3cb\") "
Oct 30 22:42:19.255655 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:19.255625    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/41d4fc04-b22d-46a5-85c5-25d46719e3cb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41d4fc04-b22d-46a5-85c5-25d46719e3cb-maas-api-xks-ca\") pod \"41d4fc04-b22d-46a5-85c5-25d46719e3cb\" (UID: \"41d4fc04-b22d-46a5-85c5-25d46719e3cb\") "
Oct 30 22:42:19.255857 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:19.255772    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41d4fc04-b22d-46a5-85c5-25d46719e3cb-kube-api-access-sxtrr\" (UniqueName: \"kubernetes.io/projected/41d4fc04-b22d-46a5-85c5-25d46719e3cb-kube-api-access-sxtrr\") pod \"41d4fc04-b22d-46a5-85c5-25d46719e3cb\" (UID: \"41d4fc04-b22d-46a5-85c5-25d46719e3cb\") "
Oct 30 22:42:19.256028 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:19.255999    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41d4fc04-b22d-46a5-85c5-25d46719e3cb-maas-api-openshift-ca" pod "41d4fc04-b22d-46a5-85c5-25d46719e3cb" (UID: "41d4fc04-b22d-46a5-85c5-25d46719e3cb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:42:19.256161 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:19.256141    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41d4fc04-b22d-46a5-85c5-25d46719e3cb-maas-api-openshift-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:42:19.258323 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:19.258293    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41d4fc04-b22d-46a5-85c5-25d46719e3cb-kube-api-access-sxtrr" pod "41d4fc04-b22d-46a5-85c5-25d46719e3cb" (UID: "41d4fc04-b22d-46a5-85c5-25d46719e3cb"). InnerVolumeSpecName "kube-api-access-sxtrr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:42:19.258627 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:19.258599    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41d4fc04-b22d-46a5-85c5-25d46719e3cb-maas-api-xks-ca" pod "41d4fc04-b22d-46a5-85c5-25d46719e3cb" (UID: "41d4fc04-b22d-46a5-85c5-25d46719e3cb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:42:19.357166 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:19.357122    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxtrr\" (UniqueName: \"kubernetes.io/projected/41d4fc04-b22d-46a5-85c5-25d46719e3cb-kube-api-access-sxtrr\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:42:19.357166 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:19.357165    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41d4fc04-b22d-46a5-85c5-25d46719e3cb-maas-api-xks-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:42:19.988556 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:19.984658    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-673c84-frswm"
Oct 30 22:42:19.997758 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:19.996593    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-673c84-frswm" event={"ID":"41d4fc04-b22d-46a5-85c5-25d46719e3cb","Type":"ContainerDied","Data":"e67fb9fcbdf8f5a9d4d144be0c3b082b39a778d2275cb8bdd47bc4ede2438cdd"}
Oct 30 22:42:19.997758 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:19.997318    5061 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e67fb9fcbdf8f5a9d4d144be0c3b082b39a778d2275cb8bdd47bc4ede2438cdd"
Oct 30 22:42:20.285551 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:20.285451    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-673c84/simulator-single-main-673c84-kserve-85c7949ff-74bxf"]
Oct 30 22:42:24.665932 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:42:24.665888    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 22:42:27.027410 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:27.027384    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-sclrt_5e74071e-0462-42b5-a05c-d90bb638478a/console-operator/2.log"
Oct 30 22:42:27.076777 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:27.076747    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-sclrt_5e74071e-0462-42b5-a05c-d90bb638478a/console-operator/2.log"
Oct 30 22:42:27.406940 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:27.406905    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-673c84-d468879f85skhz"]
Oct 30 22:42:27.516861 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:27.516825    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-673c84-d7d4fgxr4m"]
Oct 30 22:42:32.408032 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:32.407987    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-673c84-d468879f85skhz" podUID="76765e23-d419-43c7-a763-18224674780d" containerName="payload-processing" containerID="cri-o://ebb91ea58a0363984f411d3a22b8c6c9a5f1a19a8294e852a819cb47885e49f3" gracePeriod=25
Oct 30 22:42:32.517963 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:32.517898    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-673c84-d7d4fgxr4m" podUID="0ae521ec-c315-47a0-a700-d98dc9512a47" containerName="payload-pre-processing" containerID="cri-o://d9823370606408df9f2ec11b0a79f3e2264f43d0b03515f3cb8425b3e262203a" gracePeriod=25
Oct 30 22:42:35.305715 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:35.305658    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-673c84/simulator-single-main-673c84-kserve-85c7949ff-74bxf" podUID="19a817cf-0058-4865-8517-415419b98a26" containerName="main" containerID="cri-o://f17ccd799489818573edd08e085b0da3a0f29f0ef6e8fde2d4255a50be0d7271" gracePeriod=45
Oct 30 22:42:35.566058 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:35.566002    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-673c84/simulator-single-main-673c84-kserve-85c7949ff-74bxf"
Oct 30 22:42:35.631226 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:35.631195    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19a817cf-0058-4865-8517-415419b98a26-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19a817cf-0058-4865-8517-415419b98a26-tmp-dir\") pod \"19a817cf-0058-4865-8517-415419b98a26\" (UID: \"19a817cf-0058-4865-8517-415419b98a26\") "
Oct 30 22:42:35.631432 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:35.631264    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19a817cf-0058-4865-8517-415419b98a26-kube-api-access-6r984\" (UniqueName: \"kubernetes.io/projected/19a817cf-0058-4865-8517-415419b98a26-kube-api-access-6r984\") pod \"19a817cf-0058-4865-8517-415419b98a26\" (UID: \"19a817cf-0058-4865-8517-415419b98a26\") "
Oct 30 22:42:35.631432 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:35.631334    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19a817cf-0058-4865-8517-415419b98a26-dshm\" (UniqueName: \"kubernetes.io/empty-dir/19a817cf-0058-4865-8517-415419b98a26-dshm\") pod \"19a817cf-0058-4865-8517-415419b98a26\" (UID: \"19a817cf-0058-4865-8517-415419b98a26\") "
Oct 30 22:42:35.631432 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:35.631361    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19a817cf-0058-4865-8517-415419b98a26-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19a817cf-0058-4865-8517-415419b98a26-model-cache\") pod \"19a817cf-0058-4865-8517-415419b98a26\" (UID: \"19a817cf-0058-4865-8517-415419b98a26\") "
Oct 30 22:42:35.631432 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:35.631393    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19a817cf-0058-4865-8517-415419b98a26-tls-certs\" (UniqueName: \"kubernetes.io/secret/19a817cf-0058-4865-8517-415419b98a26-tls-certs\") pod \"19a817cf-0058-4865-8517-415419b98a26\" (UID: \"19a817cf-0058-4865-8517-415419b98a26\") "
Oct 30 22:42:35.631432 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:35.631425    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19a817cf-0058-4865-8517-415419b98a26-home\" (UniqueName: \"kubernetes.io/empty-dir/19a817cf-0058-4865-8517-415419b98a26-home\") pod \"19a817cf-0058-4865-8517-415419b98a26\" (UID: \"19a817cf-0058-4865-8517-415419b98a26\") "
Oct 30 22:42:35.631760 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:35.631517    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19a817cf-0058-4865-8517-415419b98a26-tmp-dir" pod "19a817cf-0058-4865-8517-415419b98a26" (UID: "19a817cf-0058-4865-8517-415419b98a26"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:42:35.631760 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:35.631699    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19a817cf-0058-4865-8517-415419b98a26-model-cache" pod "19a817cf-0058-4865-8517-415419b98a26" (UID: "19a817cf-0058-4865-8517-415419b98a26"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:42:35.631884 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:35.631765    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19a817cf-0058-4865-8517-415419b98a26-home" pod "19a817cf-0058-4865-8517-415419b98a26" (UID: "19a817cf-0058-4865-8517-415419b98a26"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:42:35.631884 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:35.631861    5061 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19a817cf-0058-4865-8517-415419b98a26-home\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:42:35.631999 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:35.631900    5061 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19a817cf-0058-4865-8517-415419b98a26-tmp-dir\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:42:35.631999 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:35.631915    5061 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19a817cf-0058-4865-8517-415419b98a26-model-cache\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:42:35.633728 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:35.633703    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19a817cf-0058-4865-8517-415419b98a26-dshm" pod "19a817cf-0058-4865-8517-415419b98a26" (UID: "19a817cf-0058-4865-8517-415419b98a26"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:42:35.633728 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:35.633718    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19a817cf-0058-4865-8517-415419b98a26-kube-api-access-6r984" pod "19a817cf-0058-4865-8517-415419b98a26" (UID: "19a817cf-0058-4865-8517-415419b98a26"). InnerVolumeSpecName "kube-api-access-6r984". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:42:35.633887 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:35.633778    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19a817cf-0058-4865-8517-415419b98a26-tls-certs" pod "19a817cf-0058-4865-8517-415419b98a26" (UID: "19a817cf-0058-4865-8517-415419b98a26"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:42:35.732922 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:35.732887    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6r984\" (UniqueName: \"kubernetes.io/projected/19a817cf-0058-4865-8517-415419b98a26-kube-api-access-6r984\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:42:35.732922 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:35.732922    5061 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19a817cf-0058-4865-8517-415419b98a26-dshm\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:42:35.733126 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:35.732932    5061 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19a817cf-0058-4865-8517-415419b98a26-tls-certs\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:42:36.072184 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:36.072150    5061 generic.go:356] "Generic (PLEG): container finished" podID="19a817cf-0058-4865-8517-415419b98a26" containerID="f17ccd799489818573edd08e085b0da3a0f29f0ef6e8fde2d4255a50be0d7271" exitCode=0
Oct 30 22:42:36.072399 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:36.072197    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-673c84/simulator-single-main-673c84-kserve-85c7949ff-74bxf"
Oct 30 22:42:36.072399 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:36.072235    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-673c84/simulator-single-main-673c84-kserve-85c7949ff-74bxf" event={"ID":"19a817cf-0058-4865-8517-415419b98a26","Type":"ContainerDied","Data":"f17ccd799489818573edd08e085b0da3a0f29f0ef6e8fde2d4255a50be0d7271"}
Oct 30 22:42:36.072399 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:36.072298    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-673c84/simulator-single-main-673c84-kserve-85c7949ff-74bxf" event={"ID":"19a817cf-0058-4865-8517-415419b98a26","Type":"ContainerDied","Data":"c1b039c48291f88d0ef0eebe7a921f2c6a9fb974c0ce2e5fe4d6fbab98c45e52"}
Oct 30 22:42:36.072399 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:36.072323    5061 scope.go:122] "RemoveContainer" containerID="f17ccd799489818573edd08e085b0da3a0f29f0ef6e8fde2d4255a50be0d7271"
Oct 30 22:42:36.083797 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:36.083766    5061 scope.go:122] "RemoveContainer" containerID="f17ccd799489818573edd08e085b0da3a0f29f0ef6e8fde2d4255a50be0d7271"
Oct 30 22:42:36.084108 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:42:36.084087    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f17ccd799489818573edd08e085b0da3a0f29f0ef6e8fde2d4255a50be0d7271\": container with ID starting with f17ccd799489818573edd08e085b0da3a0f29f0ef6e8fde2d4255a50be0d7271 not found: ID does not exist" containerID="f17ccd799489818573edd08e085b0da3a0f29f0ef6e8fde2d4255a50be0d7271"
Oct 30 22:42:36.084173 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:36.084118    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f17ccd799489818573edd08e085b0da3a0f29f0ef6e8fde2d4255a50be0d7271"} err="failed to get container status \"f17ccd799489818573edd08e085b0da3a0f29f0ef6e8fde2d4255a50be0d7271\": rpc error: code = NotFound desc = could not find container \"f17ccd799489818573edd08e085b0da3a0f29f0ef6e8fde2d4255a50be0d7271\": container with ID starting with f17ccd799489818573edd08e085b0da3a0f29f0ef6e8fde2d4255a50be0d7271 not found: ID does not exist"
Oct 30 22:42:36.246577 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:36.246543    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-673c84/simulator-single-main-673c84-kserve-85c7949ff-74bxf"]
Oct 30 22:42:36.248432 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:36.248408    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-673c84/simulator-single-main-673c84-kserve-85c7949ff-74bxf"]
Oct 30 22:42:36.298872 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:36.298843    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-673c84-frswm"]
Oct 30 22:42:36.300623 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:36.300602    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-673c84-frswm"]
Oct 30 22:42:36.408298 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:36.408213    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-673c84-d468879f85skhz" podUID="76765e23-d419-43c7-a763-18224674780d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:42:36.408691 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:36.408221    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-673c84-d7d4fgxr4m" podUID="0ae521ec-c315-47a0-a700-d98dc9512a47" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:42:36.528046 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:36.528014    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19a817cf-0058-4865-8517-415419b98a26" path="/var/lib/kubelet/pods/19a817cf-0058-4865-8517-415419b98a26/volumes"
Oct 30 22:42:36.528446 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:36.528432    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41d4fc04-b22d-46a5-85c5-25d46719e3cb" path="/var/lib/kubelet/pods/41d4fc04-b22d-46a5-85c5-25d46719e3cb/volumes"
Oct 30 22:42:46.408032 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:46.407984    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-673c84-d7d4fgxr4m" podUID="0ae521ec-c315-47a0-a700-d98dc9512a47" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:42:46.408032 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:46.407999    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-673c84-d468879f85skhz" podUID="76765e23-d419-43c7-a763-18224674780d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:42:52.690182 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:52.690147    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-673c84-d468879f85skhz"
Oct 30 22:42:52.718474 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:52.718447    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-673c84-d7d4fgxr4m"
Oct 30 22:42:52.827800 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:52.827763    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76765e23-d419-43c7-a763-18224674780d-kube-api-access-lkjj9\" (UniqueName: \"kubernetes.io/projected/76765e23-d419-43c7-a763-18224674780d-kube-api-access-lkjj9\") pod \"76765e23-d419-43c7-a763-18224674780d\" (UID: \"76765e23-d419-43c7-a763-18224674780d\") "
Oct 30 22:42:52.828000 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:52.827854    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76765e23-d419-43c7-a763-18224674780d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76765e23-d419-43c7-a763-18224674780d-plugins-config-volume\") pod \"76765e23-d419-43c7-a763-18224674780d\" (UID: \"76765e23-d419-43c7-a763-18224674780d\") "
Oct 30 22:42:52.828000 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:52.827946    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ae521ec-c315-47a0-a700-d98dc9512a47-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ae521ec-c315-47a0-a700-d98dc9512a47-plugins-config-volume\") pod \"0ae521ec-c315-47a0-a700-d98dc9512a47\" (UID: \"0ae521ec-c315-47a0-a700-d98dc9512a47\") "
Oct 30 22:42:52.828000 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:52.827982    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ae521ec-c315-47a0-a700-d98dc9512a47-kube-api-access-zsjrh\" (UniqueName: \"kubernetes.io/projected/0ae521ec-c315-47a0-a700-d98dc9512a47-kube-api-access-zsjrh\") pod \"0ae521ec-c315-47a0-a700-d98dc9512a47\" (UID: \"0ae521ec-c315-47a0-a700-d98dc9512a47\") "
Oct 30 22:42:52.828301 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:52.828273    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76765e23-d419-43c7-a763-18224674780d-plugins-config-volume" pod "76765e23-d419-43c7-a763-18224674780d" (UID: "76765e23-d419-43c7-a763-18224674780d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:42:52.828421 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:52.828393    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ae521ec-c315-47a0-a700-d98dc9512a47-plugins-config-volume" pod "0ae521ec-c315-47a0-a700-d98dc9512a47" (UID: "0ae521ec-c315-47a0-a700-d98dc9512a47"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:42:52.830083 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:52.830055    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ae521ec-c315-47a0-a700-d98dc9512a47-kube-api-access-zsjrh" pod "0ae521ec-c315-47a0-a700-d98dc9512a47" (UID: "0ae521ec-c315-47a0-a700-d98dc9512a47"). InnerVolumeSpecName "kube-api-access-zsjrh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:42:52.830185 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:52.830064    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76765e23-d419-43c7-a763-18224674780d-kube-api-access-lkjj9" pod "76765e23-d419-43c7-a763-18224674780d" (UID: "76765e23-d419-43c7-a763-18224674780d"). InnerVolumeSpecName "kube-api-access-lkjj9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:42:52.929353 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:52.929265    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkjj9\" (UniqueName: \"kubernetes.io/projected/76765e23-d419-43c7-a763-18224674780d-kube-api-access-lkjj9\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:42:52.929353 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:52.929298    5061 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76765e23-d419-43c7-a763-18224674780d-plugins-config-volume\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:42:52.929353 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:52.929308    5061 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ae521ec-c315-47a0-a700-d98dc9512a47-plugins-config-volume\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:42:52.929353 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:52.929318    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zsjrh\" (UniqueName: \"kubernetes.io/projected/0ae521ec-c315-47a0-a700-d98dc9512a47-kube-api-access-zsjrh\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:42:53.174365 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:53.174327    5061 generic.go:356] "Generic (PLEG): container finished" podID="0ae521ec-c315-47a0-a700-d98dc9512a47" containerID="d9823370606408df9f2ec11b0a79f3e2264f43d0b03515f3cb8425b3e262203a" exitCode=0
Oct 30 22:42:53.174600 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:53.174378    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-673c84-d7d4fgxr4m"
Oct 30 22:42:53.174600 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:53.174425    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-673c84-d7d4fgxr4m" event={"ID":"0ae521ec-c315-47a0-a700-d98dc9512a47","Type":"ContainerDied","Data":"d9823370606408df9f2ec11b0a79f3e2264f43d0b03515f3cb8425b3e262203a"}
Oct 30 22:42:53.174600 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:53.174508    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-673c84-d7d4fgxr4m" event={"ID":"0ae521ec-c315-47a0-a700-d98dc9512a47","Type":"ContainerDied","Data":"3b46f330f407caeb336888a25ff774b1a05da3199723069b10d8ce302d0a9fcd"}
Oct 30 22:42:53.174600 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:53.174576    5061 scope.go:122] "RemoveContainer" containerID="d9823370606408df9f2ec11b0a79f3e2264f43d0b03515f3cb8425b3e262203a"
Oct 30 22:42:53.176097 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:53.176076    5061 generic.go:356] "Generic (PLEG): container finished" podID="76765e23-d419-43c7-a763-18224674780d" containerID="ebb91ea58a0363984f411d3a22b8c6c9a5f1a19a8294e852a819cb47885e49f3" exitCode=0
Oct 30 22:42:53.176199 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:53.176129    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-673c84-d468879f85skhz"
Oct 30 22:42:53.176199 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:53.176169    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-673c84-d468879f85skhz" event={"ID":"76765e23-d419-43c7-a763-18224674780d","Type":"ContainerDied","Data":"ebb91ea58a0363984f411d3a22b8c6c9a5f1a19a8294e852a819cb47885e49f3"}
Oct 30 22:42:53.176320 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:53.176211    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-673c84-d468879f85skhz" event={"ID":"76765e23-d419-43c7-a763-18224674780d","Type":"ContainerDied","Data":"dbea6f683247a78319f6386add1d929e5417796be05595bd13b9ace2b9537ff7"}
Oct 30 22:42:53.186101 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:53.186080    5061 scope.go:122] "RemoveContainer" containerID="d9823370606408df9f2ec11b0a79f3e2264f43d0b03515f3cb8425b3e262203a"
Oct 30 22:42:53.186489 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:42:53.186451    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9823370606408df9f2ec11b0a79f3e2264f43d0b03515f3cb8425b3e262203a\": container with ID starting with d9823370606408df9f2ec11b0a79f3e2264f43d0b03515f3cb8425b3e262203a not found: ID does not exist" containerID="d9823370606408df9f2ec11b0a79f3e2264f43d0b03515f3cb8425b3e262203a"
Oct 30 22:42:53.186600 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:53.186513    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9823370606408df9f2ec11b0a79f3e2264f43d0b03515f3cb8425b3e262203a"} err="failed to get container status \"d9823370606408df9f2ec11b0a79f3e2264f43d0b03515f3cb8425b3e262203a\": rpc error: code = NotFound desc = could not find container \"d9823370606408df9f2ec11b0a79f3e2264f43d0b03515f3cb8425b3e262203a\": container with ID starting with d9823370606408df9f2ec11b0a79f3e2264f43d0b03515f3cb8425b3e262203a not found: ID does not exist"
Oct 30 22:42:53.186600 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:53.186553    5061 scope.go:122] "RemoveContainer" containerID="ebb91ea58a0363984f411d3a22b8c6c9a5f1a19a8294e852a819cb47885e49f3"
Oct 30 22:42:53.195004 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:53.194977    5061 scope.go:122] "RemoveContainer" containerID="ebb91ea58a0363984f411d3a22b8c6c9a5f1a19a8294e852a819cb47885e49f3"
Oct 30 22:42:53.195286 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:42:53.195265    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebb91ea58a0363984f411d3a22b8c6c9a5f1a19a8294e852a819cb47885e49f3\": container with ID starting with ebb91ea58a0363984f411d3a22b8c6c9a5f1a19a8294e852a819cb47885e49f3 not found: ID does not exist" containerID="ebb91ea58a0363984f411d3a22b8c6c9a5f1a19a8294e852a819cb47885e49f3"
Oct 30 22:42:53.195337 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:53.195293    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebb91ea58a0363984f411d3a22b8c6c9a5f1a19a8294e852a819cb47885e49f3"} err="failed to get container status \"ebb91ea58a0363984f411d3a22b8c6c9a5f1a19a8294e852a819cb47885e49f3\": rpc error: code = NotFound desc = could not find container \"ebb91ea58a0363984f411d3a22b8c6c9a5f1a19a8294e852a819cb47885e49f3\": container with ID starting with ebb91ea58a0363984f411d3a22b8c6c9a5f1a19a8294e852a819cb47885e49f3 not found: ID does not exist"
Oct 30 22:42:53.346179 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:53.346141    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-673c84-d7d4fgxr4m"]
Oct 30 22:42:53.398995 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:53.398963    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-673c84-d7d4fgxr4m"]
Oct 30 22:42:53.464847 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:53.464769    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-673c84-d468879f85skhz"]
Oct 30 22:42:53.466789 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:53.466768    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-673c84-d468879f85skhz"]
Oct 30 22:42:54.527595 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:54.527559    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ae521ec-c315-47a0-a700-d98dc9512a47" path="/var/lib/kubelet/pods/0ae521ec-c315-47a0-a700-d98dc9512a47/volumes"
Oct 30 22:42:54.527979 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:42:54.527952    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76765e23-d419-43c7-a763-18224674780d" path="/var/lib/kubelet/pods/76765e23-d419-43c7-a763-18224674780d/volumes"
Oct 30 22:42:54.666658 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:42:54.666625    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 22:43:24.667514 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:43:24.667405    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 22:43:54.665947 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:43:54.665911    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 22:44:24.666321 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:44:24.666285    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 22:44:54.666970 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:44:54.666883    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 22:45:24.667704 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:45:24.667667    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 22:45:43.436031 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:45:43.435996    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-b1b8bb-596f6c9h5c"]
Oct 30 22:45:43.436436 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:45:43.436425    5061 cpu_manager.go:414] "containerMap: removing container" podUID="41d4fc04-b22d-46a5-85c5-25d46719e3cb" containerName="revoke-keys"
Oct 30 22:45:43.436436 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:45:43.436436    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41d4fc04-b22d-46a5-85c5-25d46719e3cb" containerName="revoke-keys"
Oct 30 22:45:43.436513 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:45:43.436457    5061 cpu_manager.go:414] "containerMap: removing container" podUID="19a817cf-0058-4865-8517-415419b98a26" containerName="main"
Oct 30 22:45:43.436513 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:45:43.436463    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19a817cf-0058-4865-8517-415419b98a26" containerName="main"
Oct 30 22:45:43.436513 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:45:43.436478    5061 cpu_manager.go:414] "containerMap: removing container" podUID="76765e23-d419-43c7-a763-18224674780d" containerName="payload-processing"
Oct 30 22:45:43.436513 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:45:43.436484    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76765e23-d419-43c7-a763-18224674780d" containerName="payload-processing"
Oct 30 22:45:43.436513 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:45:43.436500    5061 cpu_manager.go:414] "containerMap: removing container" podUID="0ae521ec-c315-47a0-a700-d98dc9512a47" containerName="payload-pre-processing"
Oct 30 22:45:43.436513 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:45:43.436505    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ae521ec-c315-47a0-a700-d98dc9512a47" containerName="payload-pre-processing"
Oct 30 22:45:43.436745 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:45:43.436578    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ae521ec-c315-47a0-a700-d98dc9512a47" containerName="payload-pre-processing"
Oct 30 22:45:43.436745 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:45:43.436587    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="41d4fc04-b22d-46a5-85c5-25d46719e3cb" containerName="revoke-keys"
Oct 30 22:45:43.436745 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:45:43.436597    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="19a817cf-0058-4865-8517-415419b98a26" containerName="main"
Oct 30 22:45:43.436745 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:45:43.436604    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="76765e23-d419-43c7-a763-18224674780d" containerName="payload-processing"
Oct 30 22:45:43.439965 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:45:43.439946    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b1b8bb-596f6c9h5c"
Oct 30 22:45:43.498571 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:45:43.498514    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-b1b8bb\""
Oct 30 22:45:43.516027 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:45:43.515984    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4b9xg\" (UniqueName: \"kubernetes.io/projected/deb10767-aae5-4dd9-a1bd-e2bae05f88ce-kube-api-access-4b9xg\") pod \"payload-pre-processing-e2e-worker-single-main-b1b8bb-596f6c9h5c\" (UID: \"deb10767-aae5-4dd9-a1bd-e2bae05f88ce\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b1b8bb-596f6c9h5c"
Oct 30 22:45:43.516231 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:45:43.516094    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/deb10767-aae5-4dd9-a1bd-e2bae05f88ce-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-b1b8bb-596f6c9h5c\" (UID: \"deb10767-aae5-4dd9-a1bd-e2bae05f88ce\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b1b8bb-596f6c9h5c"
Oct 30 22:45:43.551249 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:45:43.551217    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-d49c53ea-dockercfg-q7j2s\""
Oct 30 22:45:43.554715 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:45:43.554679    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-b1b8bb-596f6c9h5c"]
Oct 30 22:45:43.608227 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:45:43.608196    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-b1b8bb-7d544ddbclrw4p"]
Oct 30 22:45:43.611802 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:45:43.611781    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b1b8bb-7d544ddbclrw4p"
Oct 30 22:45:43.617325 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:45:43.617302    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4b9xg\" (UniqueName: \"kubernetes.io/projected/deb10767-aae5-4dd9-a1bd-e2bae05f88ce-kube-api-access-4b9xg\") pod \"payload-pre-processing-e2e-worker-single-main-b1b8bb-596f6c9h5c\" (UID: \"deb10767-aae5-4dd9-a1bd-e2bae05f88ce\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b1b8bb-596f6c9h5c"
Oct 30 22:45:43.617476 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:45:43.617381    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/deb10767-aae5-4dd9-a1bd-e2bae05f88ce-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-b1b8bb-596f6c9h5c\" (UID: \"deb10767-aae5-4dd9-a1bd-e2bae05f88ce\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b1b8bb-596f6c9h5c"
Oct 30 22:45:43.618282 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:45:43.618126    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/deb10767-aae5-4dd9-a1bd-e2bae05f88ce-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-b1b8bb-596f6c9h5c\" (UID: \"deb10767-aae5-4dd9-a1bd-e2bae05f88ce\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b1b8bb-596f6c9h5c"
Oct 30 22:45:43.649713 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:45:43.649683    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-b1b8bb-7d544ddbclrw4p"]
Oct 30 22:45:43.677511 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:45:43.677479    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4b9xg\" (UniqueName: \"kubernetes.io/projected/deb10767-aae5-4dd9-a1bd-e2bae05f88ce-kube-api-access-4b9xg\") pod \"payload-pre-processing-e2e-worker-single-main-b1b8bb-596f6c9h5c\" (UID: \"deb10767-aae5-4dd9-a1bd-e2bae05f88ce\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b1b8bb-596f6c9h5c"
Oct 30 22:45:43.718654 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:45:43.718572    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5q4m2\" (UniqueName: \"kubernetes.io/projected/0dab0e3f-d992-4b7f-a958-7887da685867-kube-api-access-5q4m2\") pod \"payload-processing-e2e-worker-single-main-b1b8bb-7d544ddbclrw4p\" (UID: \"0dab0e3f-d992-4b7f-a958-7887da685867\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-b1b8bb-7d544ddbclrw4p"
Oct 30 22:45:43.718654 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:45:43.718619    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0dab0e3f-d992-4b7f-a958-7887da685867-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-b1b8bb-7d544ddbclrw4p\" (UID: \"0dab0e3f-d992-4b7f-a958-7887da685867\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-b1b8bb-7d544ddbclrw4p"
Oct 30 22:45:43.752412 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:45:43.752381    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b1b8bb-596f6c9h5c"
Oct 30 22:45:43.821569 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:45:43.820325    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5q4m2\" (UniqueName: \"kubernetes.io/projected/0dab0e3f-d992-4b7f-a958-7887da685867-kube-api-access-5q4m2\") pod \"payload-processing-e2e-worker-single-main-b1b8bb-7d544ddbclrw4p\" (UID: \"0dab0e3f-d992-4b7f-a958-7887da685867\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-b1b8bb-7d544ddbclrw4p"
Oct 30 22:45:43.821569 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:45:43.820386    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0dab0e3f-d992-4b7f-a958-7887da685867-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-b1b8bb-7d544ddbclrw4p\" (UID: \"0dab0e3f-d992-4b7f-a958-7887da685867\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-b1b8bb-7d544ddbclrw4p"
Oct 30 22:45:43.821569 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:45:43.821303    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0dab0e3f-d992-4b7f-a958-7887da685867-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-b1b8bb-7d544ddbclrw4p\" (UID: \"0dab0e3f-d992-4b7f-a958-7887da685867\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-b1b8bb-7d544ddbclrw4p"
Oct 30 22:45:43.881489 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:45:43.881456    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5q4m2\" (UniqueName: \"kubernetes.io/projected/0dab0e3f-d992-4b7f-a958-7887da685867-kube-api-access-5q4m2\") pod \"payload-processing-e2e-worker-single-main-b1b8bb-7d544ddbclrw4p\" (UID: \"0dab0e3f-d992-4b7f-a958-7887da685867\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-b1b8bb-7d544ddbclrw4p"
Oct 30 22:45:43.924565 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:45:43.924516    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b1b8bb-7d544ddbclrw4p"
Oct 30 22:45:44.120580 ip-10-0-130-60 kubenswrapper[5061]: W1030 22:45:44.120523    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddeb10767_aae5_4dd9_a1bd_e2bae05f88ce.slice/crio-b58d793fd1df5b713a0314f042f7a2f97206b448081022ab9b5f2c6f23226564 WatchSource:0}: Error finding container b58d793fd1df5b713a0314f042f7a2f97206b448081022ab9b5f2c6f23226564: Status 404 returned error can't find the container with id b58d793fd1df5b713a0314f042f7a2f97206b448081022ab9b5f2c6f23226564
Oct 30 22:45:44.125142 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:45:44.125115    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-b1b8bb-596f6c9h5c"]
Oct 30 22:45:44.297116 ip-10-0-130-60 kubenswrapper[5061]: W1030 22:45:44.297085    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dab0e3f_d992_4b7f_a958_7887da685867.slice/crio-5a86b705a6890e4a2f40252e2f4e0972610bc627f8d00558426ac8f94d782a40 WatchSource:0}: Error finding container 5a86b705a6890e4a2f40252e2f4e0972610bc627f8d00558426ac8f94d782a40: Status 404 returned error can't find the container with id 5a86b705a6890e4a2f40252e2f4e0972610bc627f8d00558426ac8f94d782a40
Oct 30 22:45:44.301061 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:45:44.301039    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-b1b8bb-7d544ddbclrw4p"]
Oct 30 22:45:45.118633 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:45:45.118317    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b1b8bb-7d544ddbclrw4p" event={"ID":"0dab0e3f-d992-4b7f-a958-7887da685867","Type":"ContainerStarted","Data":"c7a426e6760851e021ce2c121ff308a654e2b77687786d90489b6fafa2a2db28"}
Oct 30 22:45:45.118633 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:45:45.118378    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b1b8bb-7d544ddbclrw4p" event={"ID":"0dab0e3f-d992-4b7f-a958-7887da685867","Type":"ContainerStarted","Data":"5a86b705a6890e4a2f40252e2f4e0972610bc627f8d00558426ac8f94d782a40"}
Oct 30 22:45:45.118633 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:45:45.118407    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b1b8bb-7d544ddbclrw4p"
Oct 30 22:45:45.120188 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:45:45.120160    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b1b8bb-596f6c9h5c" event={"ID":"deb10767-aae5-4dd9-a1bd-e2bae05f88ce","Type":"ContainerStarted","Data":"4b929a7f90467335ed01bdec4a0baa3b750d919aad8ae36c62707a09d9f15555"}
Oct 30 22:45:45.120296 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:45:45.120210    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b1b8bb-596f6c9h5c" event={"ID":"deb10767-aae5-4dd9-a1bd-e2bae05f88ce","Type":"ContainerStarted","Data":"b58d793fd1df5b713a0314f042f7a2f97206b448081022ab9b5f2c6f23226564"}
Oct 30 22:45:45.120296 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:45:45.120238    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b1b8bb-596f6c9h5c"
Oct 30 22:45:45.238986 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:45:45.238942    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b1b8bb-7d544ddbclrw4p" podStartSLOduration=2.238928968 podStartE2EDuration="2.238928968s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:45:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:45:45.238733318 +0000 UTC m=+3798.851503076" watchObservedRunningTime="2026-10-30 22:45:45.238928968 +0000 UTC m=+3798.851698725"
Oct 30 22:45:45.359375 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:45:45.359308    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b1b8bb-596f6c9h5c" podStartSLOduration=2.359289217 podStartE2EDuration="2.359289217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:45:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:45:45.358631555 +0000 UTC m=+3798.971401336" watchObservedRunningTime="2026-10-30 22:45:45.359289217 +0000 UTC m=+3798.972058978"
Oct 30 22:45:54.667016 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:45:54.666976    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 22:45:56.127763 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:45:56.127732    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b1b8bb-596f6c9h5c"
Oct 30 22:45:56.128293 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:45:56.127807    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b1b8bb-7d544ddbclrw4p"
Oct 30 22:46:08.531289 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:46:08.531216    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-b1b8bb/simulator-single-main-b1b8bb-kserve-555cb456bf-jwl2j"]
Oct 30 22:46:08.534979 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:46:08.534960    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-b1b8bb/simulator-single-main-b1b8bb-kserve-555cb456bf-jwl2j"
Oct 30 22:46:08.563486 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:46:08.563447    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/818028cc-aad0-49bf-bb13-a7eef80f5ca5-tls-certs\") pod \"simulator-single-main-b1b8bb-kserve-555cb456bf-jwl2j\" (UID: \"818028cc-aad0-49bf-bb13-a7eef80f5ca5\") " pod="e2e-models-e2e-worker-single-main-b1b8bb/simulator-single-main-b1b8bb-kserve-555cb456bf-jwl2j"
Oct 30 22:46:08.563670 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:46:08.563508    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/818028cc-aad0-49bf-bb13-a7eef80f5ca5-home\") pod \"simulator-single-main-b1b8bb-kserve-555cb456bf-jwl2j\" (UID: \"818028cc-aad0-49bf-bb13-a7eef80f5ca5\") " pod="e2e-models-e2e-worker-single-main-b1b8bb/simulator-single-main-b1b8bb-kserve-555cb456bf-jwl2j"
Oct 30 22:46:08.563753 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:46:08.563666    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxtc7\" (UniqueName: \"kubernetes.io/projected/818028cc-aad0-49bf-bb13-a7eef80f5ca5-kube-api-access-bxtc7\") pod \"simulator-single-main-b1b8bb-kserve-555cb456bf-jwl2j\" (UID: \"818028cc-aad0-49bf-bb13-a7eef80f5ca5\") " pod="e2e-models-e2e-worker-single-main-b1b8bb/simulator-single-main-b1b8bb-kserve-555cb456bf-jwl2j"
Oct 30 22:46:08.563810 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:46:08.563756    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/818028cc-aad0-49bf-bb13-a7eef80f5ca5-dshm\") pod \"simulator-single-main-b1b8bb-kserve-555cb456bf-jwl2j\" (UID: \"818028cc-aad0-49bf-bb13-a7eef80f5ca5\") " pod="e2e-models-e2e-worker-single-main-b1b8bb/simulator-single-main-b1b8bb-kserve-555cb456bf-jwl2j"
Oct 30 22:46:08.563895 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:46:08.563870    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/818028cc-aad0-49bf-bb13-a7eef80f5ca5-tmp-dir\") pod \"simulator-single-main-b1b8bb-kserve-555cb456bf-jwl2j\" (UID: \"818028cc-aad0-49bf-bb13-a7eef80f5ca5\") " pod="e2e-models-e2e-worker-single-main-b1b8bb/simulator-single-main-b1b8bb-kserve-555cb456bf-jwl2j"
Oct 30 22:46:08.564080 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:46:08.564059    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/818028cc-aad0-49bf-bb13-a7eef80f5ca5-model-cache\") pod \"simulator-single-main-b1b8bb-kserve-555cb456bf-jwl2j\" (UID: \"818028cc-aad0-49bf-bb13-a7eef80f5ca5\") " pod="e2e-models-e2e-worker-single-main-b1b8bb/simulator-single-main-b1b8bb-kserve-555cb456bf-jwl2j"
Oct 30 22:46:08.606564 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:46:08.606519    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-b1b8bb/simulator-single-main-b1b8bb-kserve-555cb456bf-jwl2j"]
Oct 30 22:46:08.644651 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:46:08.644621    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-b1b8bb\"/\"kube-root-ca.crt\""
Oct 30 22:46:08.644651 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:46:08.644621    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-b1b8bb\"/\"openshift-service-ca.crt\""
Oct 30 22:46:08.644860 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:46:08.644651    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-b1b8bb\"/\"default-dockercfg-c98xp\""
Oct 30 22:46:08.644860 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:46:08.644677    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-b1b8bb\"/\"simulator-single-main-b1b8bb-kserve-self-signed-certs\""
Oct 30 22:46:08.665321 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:46:08.665296    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxtc7\" (UniqueName: \"kubernetes.io/projected/818028cc-aad0-49bf-bb13-a7eef80f5ca5-kube-api-access-bxtc7\") pod \"simulator-single-main-b1b8bb-kserve-555cb456bf-jwl2j\" (UID: \"818028cc-aad0-49bf-bb13-a7eef80f5ca5\") " pod="e2e-models-e2e-worker-single-main-b1b8bb/simulator-single-main-b1b8bb-kserve-555cb456bf-jwl2j"
Oct 30 22:46:08.665484 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:46:08.665329    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/818028cc-aad0-49bf-bb13-a7eef80f5ca5-dshm\") pod \"simulator-single-main-b1b8bb-kserve-555cb456bf-jwl2j\" (UID: \"818028cc-aad0-49bf-bb13-a7eef80f5ca5\") " pod="e2e-models-e2e-worker-single-main-b1b8bb/simulator-single-main-b1b8bb-kserve-555cb456bf-jwl2j"
Oct 30 22:46:08.665484 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:46:08.665363    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/818028cc-aad0-49bf-bb13-a7eef80f5ca5-tmp-dir\") pod \"simulator-single-main-b1b8bb-kserve-555cb456bf-jwl2j\" (UID: \"818028cc-aad0-49bf-bb13-a7eef80f5ca5\") " pod="e2e-models-e2e-worker-single-main-b1b8bb/simulator-single-main-b1b8bb-kserve-555cb456bf-jwl2j"
Oct 30 22:46:08.665484 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:46:08.665398    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/818028cc-aad0-49bf-bb13-a7eef80f5ca5-model-cache\") pod \"simulator-single-main-b1b8bb-kserve-555cb456bf-jwl2j\" (UID: \"818028cc-aad0-49bf-bb13-a7eef80f5ca5\") " pod="e2e-models-e2e-worker-single-main-b1b8bb/simulator-single-main-b1b8bb-kserve-555cb456bf-jwl2j"
Oct 30 22:46:08.665740 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:46:08.665716    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/818028cc-aad0-49bf-bb13-a7eef80f5ca5-tls-certs\") pod \"simulator-single-main-b1b8bb-kserve-555cb456bf-jwl2j\" (UID: \"818028cc-aad0-49bf-bb13-a7eef80f5ca5\") " pod="e2e-models-e2e-worker-single-main-b1b8bb/simulator-single-main-b1b8bb-kserve-555cb456bf-jwl2j"
Oct 30 22:46:08.665972 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:46:08.665873    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/818028cc-aad0-49bf-bb13-a7eef80f5ca5-home\") pod \"simulator-single-main-b1b8bb-kserve-555cb456bf-jwl2j\" (UID: \"818028cc-aad0-49bf-bb13-a7eef80f5ca5\") " pod="e2e-models-e2e-worker-single-main-b1b8bb/simulator-single-main-b1b8bb-kserve-555cb456bf-jwl2j"
Oct 30 22:46:08.665972 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:46:08.665917    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/818028cc-aad0-49bf-bb13-a7eef80f5ca5-tmp-dir\") pod \"simulator-single-main-b1b8bb-kserve-555cb456bf-jwl2j\" (UID: \"818028cc-aad0-49bf-bb13-a7eef80f5ca5\") " pod="e2e-models-e2e-worker-single-main-b1b8bb/simulator-single-main-b1b8bb-kserve-555cb456bf-jwl2j"
Oct 30 22:46:08.666162 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:46:08.665965    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/818028cc-aad0-49bf-bb13-a7eef80f5ca5-model-cache\") pod \"simulator-single-main-b1b8bb-kserve-555cb456bf-jwl2j\" (UID: \"818028cc-aad0-49bf-bb13-a7eef80f5ca5\") " pod="e2e-models-e2e-worker-single-main-b1b8bb/simulator-single-main-b1b8bb-kserve-555cb456bf-jwl2j"
Oct 30 22:46:08.667229 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:46:08.666401    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/818028cc-aad0-49bf-bb13-a7eef80f5ca5-home\") pod \"simulator-single-main-b1b8bb-kserve-555cb456bf-jwl2j\" (UID: \"818028cc-aad0-49bf-bb13-a7eef80f5ca5\") " pod="e2e-models-e2e-worker-single-main-b1b8bb/simulator-single-main-b1b8bb-kserve-555cb456bf-jwl2j"
Oct 30 22:46:08.668451 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:46:08.668429    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/818028cc-aad0-49bf-bb13-a7eef80f5ca5-dshm\") pod \"simulator-single-main-b1b8bb-kserve-555cb456bf-jwl2j\" (UID: \"818028cc-aad0-49bf-bb13-a7eef80f5ca5\") " pod="e2e-models-e2e-worker-single-main-b1b8bb/simulator-single-main-b1b8bb-kserve-555cb456bf-jwl2j"
Oct 30 22:46:08.668992 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:46:08.668971    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/818028cc-aad0-49bf-bb13-a7eef80f5ca5-tls-certs\") pod \"simulator-single-main-b1b8bb-kserve-555cb456bf-jwl2j\" (UID: \"818028cc-aad0-49bf-bb13-a7eef80f5ca5\") " pod="e2e-models-e2e-worker-single-main-b1b8bb/simulator-single-main-b1b8bb-kserve-555cb456bf-jwl2j"
Oct 30 22:46:08.726461 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:46:08.726422    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxtc7\" (UniqueName: \"kubernetes.io/projected/818028cc-aad0-49bf-bb13-a7eef80f5ca5-kube-api-access-bxtc7\") pod \"simulator-single-main-b1b8bb-kserve-555cb456bf-jwl2j\" (UID: \"818028cc-aad0-49bf-bb13-a7eef80f5ca5\") " pod="e2e-models-e2e-worker-single-main-b1b8bb/simulator-single-main-b1b8bb-kserve-555cb456bf-jwl2j"
Oct 30 22:46:08.846748 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:46:08.846714    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-b1b8bb/simulator-single-main-b1b8bb-kserve-555cb456bf-jwl2j"
Oct 30 22:46:09.213474 ip-10-0-130-60 kubenswrapper[5061]: W1030 22:46:09.213444    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod818028cc_aad0_49bf_bb13_a7eef80f5ca5.slice/crio-12571cd1d3719eaf92f76c69f3ccec9d9dd52eac4bed544e96115381008a6946 WatchSource:0}: Error finding container 12571cd1d3719eaf92f76c69f3ccec9d9dd52eac4bed544e96115381008a6946: Status 404 returned error can't find the container with id 12571cd1d3719eaf92f76c69f3ccec9d9dd52eac4bed544e96115381008a6946
Oct 30 22:46:09.217337 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:46:09.217310    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-b1b8bb/simulator-single-main-b1b8bb-kserve-555cb456bf-jwl2j"]
Oct 30 22:46:09.244820 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:46:09.244796    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-b1b8bb/simulator-single-main-b1b8bb-kserve-555cb456bf-jwl2j" event={"ID":"818028cc-aad0-49bf-bb13-a7eef80f5ca5","Type":"ContainerStarted","Data":"12571cd1d3719eaf92f76c69f3ccec9d9dd52eac4bed544e96115381008a6946"}
Oct 30 22:46:10.251600 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:46:10.251558    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-b1b8bb/simulator-single-main-b1b8bb-kserve-555cb456bf-jwl2j" event={"ID":"818028cc-aad0-49bf-bb13-a7eef80f5ca5","Type":"ContainerStarted","Data":"4d62194c1741917895d2baeaad40e78cf304bf262326cc3f3914404be734429b"}
Oct 30 22:46:10.375924 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:46:10.375854    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-b1b8bb/simulator-single-main-b1b8bb-kserve-555cb456bf-jwl2j" podStartSLOduration=2.375835116 podStartE2EDuration="2.375835116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:46:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:46:10.374000793 +0000 UTC m=+3823.986770594" watchObservedRunningTime="2026-10-30 22:46:10.375835116 +0000 UTC m=+3823.988604874"
Oct 30 22:46:10.847694 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:46:10.847654    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-b1b8bb/simulator-single-main-b1b8bb-kserve-555cb456bf-jwl2j"
Oct 30 22:46:18.847163 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:46:18.847126    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-b1b8bb/simulator-single-main-b1b8bb-kserve-555cb456bf-jwl2j"
Oct 30 22:46:18.859815 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:46:18.859786    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-b1b8bb/simulator-single-main-b1b8bb-kserve-555cb456bf-jwl2j"
Oct 30 22:46:19.309141 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:46:19.309112    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-b1b8bb/simulator-single-main-b1b8bb-kserve-555cb456bf-jwl2j"
Oct 30 22:46:24.665395 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:46:24.665346    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 22:46:53.662039 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:46:53.662006    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1efcb985-hchct"]
Oct 30 22:46:53.665933 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:46:53.665916    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1efcb985-hchct"
Oct 30 22:46:53.707938 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:46:53.707912    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db90a23a-4a63-490b-8e3c-4fe6e50f1df7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1efcb985-hchct\" (UID: \"db90a23a-4a63-490b-8e3c-4fe6e50f1df7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1efcb985-hchct"
Oct 30 22:46:53.708064 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:46:53.707952    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db90a23a-4a63-490b-8e3c-4fe6e50f1df7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1efcb985-hchct\" (UID: \"db90a23a-4a63-490b-8e3c-4fe6e50f1df7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1efcb985-hchct"
Oct 30 22:46:53.708064 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:46:53.708017    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prbfj\" (UniqueName: \"kubernetes.io/projected/db90a23a-4a63-490b-8e3c-4fe6e50f1df7-kube-api-access-prbfj\") pod \"maas-api-revoke-sub-1efcb985-hchct\" (UID: \"db90a23a-4a63-490b-8e3c-4fe6e50f1df7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1efcb985-hchct"
Oct 30 22:46:53.722272 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:46:53.722247    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:46:53.775188 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:46:53.775155    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:46:53.775332 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:46:53.775203    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:46:53.775332 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:46:53.775204    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gdmfg\""
Oct 30 22:46:53.777442 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:46:53.777423    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1efcb985-hchct"]
Oct 30 22:46:53.809254 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:46:53.809233    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db90a23a-4a63-490b-8e3c-4fe6e50f1df7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1efcb985-hchct\" (UID: \"db90a23a-4a63-490b-8e3c-4fe6e50f1df7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1efcb985-hchct"
Oct 30 22:46:53.809356 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:46:53.809284    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prbfj\" (UniqueName: \"kubernetes.io/projected/db90a23a-4a63-490b-8e3c-4fe6e50f1df7-kube-api-access-prbfj\") pod \"maas-api-revoke-sub-1efcb985-hchct\" (UID: \"db90a23a-4a63-490b-8e3c-4fe6e50f1df7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1efcb985-hchct"
Oct 30 22:46:53.809356 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:46:53.809350    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db90a23a-4a63-490b-8e3c-4fe6e50f1df7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1efcb985-hchct\" (UID: \"db90a23a-4a63-490b-8e3c-4fe6e50f1df7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1efcb985-hchct"
Oct 30 22:46:53.809969 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:46:53.809948    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db90a23a-4a63-490b-8e3c-4fe6e50f1df7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1efcb985-hchct\" (UID: \"db90a23a-4a63-490b-8e3c-4fe6e50f1df7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1efcb985-hchct"
Oct 30 22:46:53.811736 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:46:53.811712    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db90a23a-4a63-490b-8e3c-4fe6e50f1df7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1efcb985-hchct\" (UID: \"db90a23a-4a63-490b-8e3c-4fe6e50f1df7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1efcb985-hchct"
Oct 30 22:46:53.869618 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:46:53.869595    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prbfj\" (UniqueName: \"kubernetes.io/projected/db90a23a-4a63-490b-8e3c-4fe6e50f1df7-kube-api-access-prbfj\") pod \"maas-api-revoke-sub-1efcb985-hchct\" (UID: \"db90a23a-4a63-490b-8e3c-4fe6e50f1df7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1efcb985-hchct"
Oct 30 22:46:53.978284 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:46:53.978203    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1efcb985-hchct"
Oct 30 22:46:54.344339 ip-10-0-130-60 kubenswrapper[5061]: W1030 22:46:54.344304    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb90a23a_4a63_490b_8e3c_4fe6e50f1df7.slice/crio-8aa1d7137cec286bfd238ad597294929619cc3104c964a8163b802fff581175e WatchSource:0}: Error finding container 8aa1d7137cec286bfd238ad597294929619cc3104c964a8163b802fff581175e: Status 404 returned error can't find the container with id 8aa1d7137cec286bfd238ad597294929619cc3104c964a8163b802fff581175e
Oct 30 22:46:54.348473 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:46:54.348444    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1efcb985-hchct"]
Oct 30 22:46:54.488403 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:46:54.488361    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1efcb985-hchct" event={"ID":"db90a23a-4a63-490b-8e3c-4fe6e50f1df7","Type":"ContainerStarted","Data":"8aa1d7137cec286bfd238ad597294929619cc3104c964a8163b802fff581175e"}
Oct 30 22:46:54.666160 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:46:54.666123    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 22:46:55.495308 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:46:55.495225    5061 generic.go:356] "Generic (PLEG): container finished" podID="db90a23a-4a63-490b-8e3c-4fe6e50f1df7" containerID="f9c4520b670857a0c392316c8e05950ef322223906e91407df16948e7ddaf64f" exitCode=0
Oct 30 22:46:55.495308 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:46:55.495266    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1efcb985-hchct" event={"ID":"db90a23a-4a63-490b-8e3c-4fe6e50f1df7","Type":"ContainerDied","Data":"f9c4520b670857a0c392316c8e05950ef322223906e91407df16948e7ddaf64f"}
Oct 30 22:46:56.765909 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:46:56.765885    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1efcb985-hchct"
Oct 30 22:46:56.945278 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:46:56.945184    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db90a23a-4a63-490b-8e3c-4fe6e50f1df7-kube-api-access-prbfj\" (UniqueName: \"kubernetes.io/projected/db90a23a-4a63-490b-8e3c-4fe6e50f1df7-kube-api-access-prbfj\") pod \"db90a23a-4a63-490b-8e3c-4fe6e50f1df7\" (UID: \"db90a23a-4a63-490b-8e3c-4fe6e50f1df7\") "
Oct 30 22:46:56.945278 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:46:56.945282    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db90a23a-4a63-490b-8e3c-4fe6e50f1df7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db90a23a-4a63-490b-8e3c-4fe6e50f1df7-maas-api-openshift-ca\") pod \"db90a23a-4a63-490b-8e3c-4fe6e50f1df7\" (UID: \"db90a23a-4a63-490b-8e3c-4fe6e50f1df7\") "
Oct 30 22:46:56.945506 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:46:56.945481    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db90a23a-4a63-490b-8e3c-4fe6e50f1df7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db90a23a-4a63-490b-8e3c-4fe6e50f1df7-maas-api-xks-ca\") pod \"db90a23a-4a63-490b-8e3c-4fe6e50f1df7\" (UID: \"db90a23a-4a63-490b-8e3c-4fe6e50f1df7\") "
Oct 30 22:46:56.945632 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:46:56.945606    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db90a23a-4a63-490b-8e3c-4fe6e50f1df7-maas-api-openshift-ca" pod "db90a23a-4a63-490b-8e3c-4fe6e50f1df7" (UID: "db90a23a-4a63-490b-8e3c-4fe6e50f1df7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:46:56.945978 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:46:56.945940    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db90a23a-4a63-490b-8e3c-4fe6e50f1df7-maas-api-openshift-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:46:56.947391 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:46:56.947365    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db90a23a-4a63-490b-8e3c-4fe6e50f1df7-kube-api-access-prbfj" pod "db90a23a-4a63-490b-8e3c-4fe6e50f1df7" (UID: "db90a23a-4a63-490b-8e3c-4fe6e50f1df7"). InnerVolumeSpecName "kube-api-access-prbfj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:46:56.947487 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:46:56.947448    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db90a23a-4a63-490b-8e3c-4fe6e50f1df7-maas-api-xks-ca" pod "db90a23a-4a63-490b-8e3c-4fe6e50f1df7" (UID: "db90a23a-4a63-490b-8e3c-4fe6e50f1df7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:46:57.047479 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:46:57.047442    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db90a23a-4a63-490b-8e3c-4fe6e50f1df7-maas-api-xks-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:46:57.047479 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:46:57.047476    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prbfj\" (UniqueName: \"kubernetes.io/projected/db90a23a-4a63-490b-8e3c-4fe6e50f1df7-kube-api-access-prbfj\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:46:57.604144 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:46:57.604108    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1efcb985-hchct" event={"ID":"db90a23a-4a63-490b-8e3c-4fe6e50f1df7","Type":"ContainerDied","Data":"8aa1d7137cec286bfd238ad597294929619cc3104c964a8163b802fff581175e"}
Oct 30 22:46:57.604304 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:46:57.604156    5061 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8aa1d7137cec286bfd238ad597294929619cc3104c964a8163b802fff581175e"
Oct 30 22:46:57.604304 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:46:57.604118    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1efcb985-hchct"
Oct 30 22:47:05.217736 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:05.217700    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b1b8bb-2nn8p"]
Oct 30 22:47:05.218225 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:05.218123    5061 cpu_manager.go:414] "containerMap: removing container" podUID="db90a23a-4a63-490b-8e3c-4fe6e50f1df7" containerName="revoke-keys"
Oct 30 22:47:05.218225 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:05.218133    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db90a23a-4a63-490b-8e3c-4fe6e50f1df7" containerName="revoke-keys"
Oct 30 22:47:05.218225 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:05.218224    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="db90a23a-4a63-490b-8e3c-4fe6e50f1df7" containerName="revoke-keys"
Oct 30 22:47:05.221368 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:05.221351    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b1b8bb-2nn8p"
Oct 30 22:47:05.276840 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:05.276808    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:47:05.324784 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:05.324752    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5smh9\" (UniqueName: \"kubernetes.io/projected/5733b960-353a-4818-93c6-f1c02943ecb7-kube-api-access-5smh9\") pod \"maas-api-revoke-keys-e2e-worker-single-main-b1b8bb-2nn8p\" (UID: \"5733b960-353a-4818-93c6-f1c02943ecb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b1b8bb-2nn8p"
Oct 30 22:47:05.324943 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:05.324804    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5733b960-353a-4818-93c6-f1c02943ecb7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-b1b8bb-2nn8p\" (UID: \"5733b960-353a-4818-93c6-f1c02943ecb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b1b8bb-2nn8p"
Oct 30 22:47:05.324943 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:05.324923    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5733b960-353a-4818-93c6-f1c02943ecb7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-b1b8bb-2nn8p\" (UID: \"5733b960-353a-4818-93c6-f1c02943ecb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b1b8bb-2nn8p"
Oct 30 22:47:05.329600 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:05.329573    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:47:05.329738 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:05.329656    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:47:05.329825 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:05.329806    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gdmfg\""
Oct 30 22:47:05.331867 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:05.331848    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b1b8bb-2nn8p"]
Oct 30 22:47:05.426062 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:05.426024    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5733b960-353a-4818-93c6-f1c02943ecb7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-b1b8bb-2nn8p\" (UID: \"5733b960-353a-4818-93c6-f1c02943ecb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b1b8bb-2nn8p"
Oct 30 22:47:05.426285 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:05.426128    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5smh9\" (UniqueName: \"kubernetes.io/projected/5733b960-353a-4818-93c6-f1c02943ecb7-kube-api-access-5smh9\") pod \"maas-api-revoke-keys-e2e-worker-single-main-b1b8bb-2nn8p\" (UID: \"5733b960-353a-4818-93c6-f1c02943ecb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b1b8bb-2nn8p"
Oct 30 22:47:05.426285 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:05.426190    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5733b960-353a-4818-93c6-f1c02943ecb7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-b1b8bb-2nn8p\" (UID: \"5733b960-353a-4818-93c6-f1c02943ecb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b1b8bb-2nn8p"
Oct 30 22:47:05.426931 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:05.426904    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5733b960-353a-4818-93c6-f1c02943ecb7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-b1b8bb-2nn8p\" (UID: \"5733b960-353a-4818-93c6-f1c02943ecb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b1b8bb-2nn8p"
Oct 30 22:47:05.428408 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:05.428388    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5733b960-353a-4818-93c6-f1c02943ecb7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-b1b8bb-2nn8p\" (UID: \"5733b960-353a-4818-93c6-f1c02943ecb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b1b8bb-2nn8p"
Oct 30 22:47:05.485881 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:05.485818    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5smh9\" (UniqueName: \"kubernetes.io/projected/5733b960-353a-4818-93c6-f1c02943ecb7-kube-api-access-5smh9\") pod \"maas-api-revoke-keys-e2e-worker-single-main-b1b8bb-2nn8p\" (UID: \"5733b960-353a-4818-93c6-f1c02943ecb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b1b8bb-2nn8p"
Oct 30 22:47:05.534634 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:05.534606    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b1b8bb-2nn8p"
Oct 30 22:47:05.903696 ip-10-0-130-60 kubenswrapper[5061]: W1030 22:47:05.903662    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5733b960_353a_4818_93c6_f1c02943ecb7.slice/crio-315d988448c36e1d14a960616149b370472fabefbbe08dcc686deccb935c7d57 WatchSource:0}: Error finding container 315d988448c36e1d14a960616149b370472fabefbbe08dcc686deccb935c7d57: Status 404 returned error can't find the container with id 315d988448c36e1d14a960616149b370472fabefbbe08dcc686deccb935c7d57
Oct 30 22:47:05.909187 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:05.909160    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b1b8bb-2nn8p"]
Oct 30 22:47:06.653184 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:06.653145    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b1b8bb-2nn8p" event={"ID":"5733b960-353a-4818-93c6-f1c02943ecb7","Type":"ContainerStarted","Data":"4ba0a33a5874f6cfeb648631d1628279b99f53722dba4e657e7979a188ef6b39"}
Oct 30 22:47:06.653680 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:06.653198    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b1b8bb-2nn8p" event={"ID":"5733b960-353a-4818-93c6-f1c02943ecb7","Type":"ContainerStarted","Data":"315d988448c36e1d14a960616149b370472fabefbbe08dcc686deccb935c7d57"}
Oct 30 22:47:06.774513 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:06.774461    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b1b8bb-2nn8p" podStartSLOduration=1.7744498100000001 podStartE2EDuration="1.77444981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:47:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:47:06.772264426 +0000 UTC m=+3880.385034184" watchObservedRunningTime="2026-10-30 22:47:06.77444981 +0000 UTC m=+3880.387219569"
Oct 30 22:47:07.659523 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:07.659488    5061 generic.go:356] "Generic (PLEG): container finished" podID="5733b960-353a-4818-93c6-f1c02943ecb7" containerID="4ba0a33a5874f6cfeb648631d1628279b99f53722dba4e657e7979a188ef6b39" exitCode=0
Oct 30 22:47:07.659941 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:07.659573    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b1b8bb-2nn8p" event={"ID":"5733b960-353a-4818-93c6-f1c02943ecb7","Type":"ContainerDied","Data":"4ba0a33a5874f6cfeb648631d1628279b99f53722dba4e657e7979a188ef6b39"}
Oct 30 22:47:08.053222 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:08.053189    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac4883ef-5hk5q"]
Oct 30 22:47:08.055132 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:08.055109    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac4883ef-5hk5q"]
Oct 30 22:47:08.527426 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:08.527348    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5acee650-6028-4568-b758-fd9f45672ff4" path="/var/lib/kubelet/pods/5acee650-6028-4568-b758-fd9f45672ff4/volumes"
Oct 30 22:47:08.804256 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:08.804233    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b1b8bb-2nn8p"
Oct 30 22:47:08.860550 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:08.860501    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5733b960-353a-4818-93c6-f1c02943ecb7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5733b960-353a-4818-93c6-f1c02943ecb7-maas-api-xks-ca\") pod \"5733b960-353a-4818-93c6-f1c02943ecb7\" (UID: \"5733b960-353a-4818-93c6-f1c02943ecb7\") "
Oct 30 22:47:08.860697 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:08.860571    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5733b960-353a-4818-93c6-f1c02943ecb7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5733b960-353a-4818-93c6-f1c02943ecb7-maas-api-openshift-ca\") pod \"5733b960-353a-4818-93c6-f1c02943ecb7\" (UID: \"5733b960-353a-4818-93c6-f1c02943ecb7\") "
Oct 30 22:47:08.860697 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:08.860604    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5733b960-353a-4818-93c6-f1c02943ecb7-kube-api-access-5smh9\" (UniqueName: \"kubernetes.io/projected/5733b960-353a-4818-93c6-f1c02943ecb7-kube-api-access-5smh9\") pod \"5733b960-353a-4818-93c6-f1c02943ecb7\" (UID: \"5733b960-353a-4818-93c6-f1c02943ecb7\") "
Oct 30 22:47:08.860912 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:08.860887    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5733b960-353a-4818-93c6-f1c02943ecb7-maas-api-openshift-ca" pod "5733b960-353a-4818-93c6-f1c02943ecb7" (UID: "5733b960-353a-4818-93c6-f1c02943ecb7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:47:08.861002 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:08.860987    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5733b960-353a-4818-93c6-f1c02943ecb7-maas-api-openshift-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:08.862612 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:08.862591    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5733b960-353a-4818-93c6-f1c02943ecb7-maas-api-xks-ca" pod "5733b960-353a-4818-93c6-f1c02943ecb7" (UID: "5733b960-353a-4818-93c6-f1c02943ecb7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:47:08.862780 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:08.862758    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5733b960-353a-4818-93c6-f1c02943ecb7-kube-api-access-5smh9" pod "5733b960-353a-4818-93c6-f1c02943ecb7" (UID: "5733b960-353a-4818-93c6-f1c02943ecb7"). InnerVolumeSpecName "kube-api-access-5smh9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:47:08.961929 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:08.961889    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5733b960-353a-4818-93c6-f1c02943ecb7-maas-api-xks-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:08.961929 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:08.961919    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5smh9\" (UniqueName: \"kubernetes.io/projected/5733b960-353a-4818-93c6-f1c02943ecb7-kube-api-access-5smh9\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:09.670492 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:09.670461    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b1b8bb-2nn8p"
Oct 30 22:47:09.670680 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:09.670460    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b1b8bb-2nn8p" event={"ID":"5733b960-353a-4818-93c6-f1c02943ecb7","Type":"ContainerDied","Data":"315d988448c36e1d14a960616149b370472fabefbbe08dcc686deccb935c7d57"}
Oct 30 22:47:09.670680 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:09.670592    5061 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="315d988448c36e1d14a960616149b370472fabefbbe08dcc686deccb935c7d57"
Oct 30 22:47:09.711207 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:09.711177    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-b1b8bb/simulator-single-main-b1b8bb-kserve-555cb456bf-jwl2j"]
Oct 30 22:47:16.616609 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:16.616572    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-b1b8bb-7d544ddbclrw4p"]
Oct 30 22:47:16.726924 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:16.726891    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-b1b8bb-596f6c9h5c"]
Oct 30 22:47:21.418126 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:21.418093    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b1b8bb-2nn8p"]
Oct 30 22:47:21.452387 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:21.452360    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b1b8bb-2nn8p"]
Oct 30 22:47:21.617915 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:21.617859    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b1b8bb-7d544ddbclrw4p" podUID="0dab0e3f-d992-4b7f-a958-7887da685867" containerName="payload-processing" containerID="cri-o://c7a426e6760851e021ce2c121ff308a654e2b77687786d90489b6fafa2a2db28" gracePeriod=25
Oct 30 22:47:21.727670 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:21.727587    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b1b8bb-596f6c9h5c" podUID="deb10767-aae5-4dd9-a1bd-e2bae05f88ce" containerName="payload-pre-processing" containerID="cri-o://4b929a7f90467335ed01bdec4a0baa3b750d919aad8ae36c62707a09d9f15555" gracePeriod=25
Oct 30 22:47:22.528693 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:22.528661    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5733b960-353a-4818-93c6-f1c02943ecb7" path="/var/lib/kubelet/pods/5733b960-353a-4818-93c6-f1c02943ecb7/volumes"
Oct 30 22:47:24.666235 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:47:24.666197    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 22:47:24.731177 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:24.731125    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-b1b8bb/simulator-single-main-b1b8bb-kserve-555cb456bf-jwl2j" podUID="818028cc-aad0-49bf-bb13-a7eef80f5ca5" containerName="main" containerID="cri-o://4d62194c1741917895d2baeaad40e78cf304bf262326cc3f3914404be734429b" gracePeriod=45
Oct 30 22:47:24.986803 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:24.986779    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-b1b8bb/simulator-single-main-b1b8bb-kserve-555cb456bf-jwl2j"
Oct 30 22:47:25.121723 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:25.121692    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/818028cc-aad0-49bf-bb13-a7eef80f5ca5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/818028cc-aad0-49bf-bb13-a7eef80f5ca5-dshm\") pod \"818028cc-aad0-49bf-bb13-a7eef80f5ca5\" (UID: \"818028cc-aad0-49bf-bb13-a7eef80f5ca5\") "
Oct 30 22:47:25.121927 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:25.121738    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/818028cc-aad0-49bf-bb13-a7eef80f5ca5-kube-api-access-bxtc7\" (UniqueName: \"kubernetes.io/projected/818028cc-aad0-49bf-bb13-a7eef80f5ca5-kube-api-access-bxtc7\") pod \"818028cc-aad0-49bf-bb13-a7eef80f5ca5\" (UID: \"818028cc-aad0-49bf-bb13-a7eef80f5ca5\") "
Oct 30 22:47:25.121927 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:25.121774    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/818028cc-aad0-49bf-bb13-a7eef80f5ca5-home\" (UniqueName: \"kubernetes.io/empty-dir/818028cc-aad0-49bf-bb13-a7eef80f5ca5-home\") pod \"818028cc-aad0-49bf-bb13-a7eef80f5ca5\" (UID: \"818028cc-aad0-49bf-bb13-a7eef80f5ca5\") "
Oct 30 22:47:25.121927 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:25.121805    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/818028cc-aad0-49bf-bb13-a7eef80f5ca5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/818028cc-aad0-49bf-bb13-a7eef80f5ca5-tmp-dir\") pod \"818028cc-aad0-49bf-bb13-a7eef80f5ca5\" (UID: \"818028cc-aad0-49bf-bb13-a7eef80f5ca5\") "
Oct 30 22:47:25.121927 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:25.121878    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/818028cc-aad0-49bf-bb13-a7eef80f5ca5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/818028cc-aad0-49bf-bb13-a7eef80f5ca5-model-cache\") pod \"818028cc-aad0-49bf-bb13-a7eef80f5ca5\" (UID: \"818028cc-aad0-49bf-bb13-a7eef80f5ca5\") "
Oct 30 22:47:25.122223 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:25.121934    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/818028cc-aad0-49bf-bb13-a7eef80f5ca5-tls-certs\" (UniqueName: \"kubernetes.io/secret/818028cc-aad0-49bf-bb13-a7eef80f5ca5-tls-certs\") pod \"818028cc-aad0-49bf-bb13-a7eef80f5ca5\" (UID: \"818028cc-aad0-49bf-bb13-a7eef80f5ca5\") "
Oct 30 22:47:25.122337 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:25.122300    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/818028cc-aad0-49bf-bb13-a7eef80f5ca5-tmp-dir" pod "818028cc-aad0-49bf-bb13-a7eef80f5ca5" (UID: "818028cc-aad0-49bf-bb13-a7eef80f5ca5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:47:25.122416 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:25.122313    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/818028cc-aad0-49bf-bb13-a7eef80f5ca5-model-cache" pod "818028cc-aad0-49bf-bb13-a7eef80f5ca5" (UID: "818028cc-aad0-49bf-bb13-a7eef80f5ca5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:47:25.122564 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:25.122504    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/818028cc-aad0-49bf-bb13-a7eef80f5ca5-home" pod "818028cc-aad0-49bf-bb13-a7eef80f5ca5" (UID: "818028cc-aad0-49bf-bb13-a7eef80f5ca5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:47:25.124668 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:25.124643    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/818028cc-aad0-49bf-bb13-a7eef80f5ca5-tls-certs" pod "818028cc-aad0-49bf-bb13-a7eef80f5ca5" (UID: "818028cc-aad0-49bf-bb13-a7eef80f5ca5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:47:25.124668 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:25.124649    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/818028cc-aad0-49bf-bb13-a7eef80f5ca5-dshm" pod "818028cc-aad0-49bf-bb13-a7eef80f5ca5" (UID: "818028cc-aad0-49bf-bb13-a7eef80f5ca5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:47:25.124809 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:25.124694    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/818028cc-aad0-49bf-bb13-a7eef80f5ca5-kube-api-access-bxtc7" pod "818028cc-aad0-49bf-bb13-a7eef80f5ca5" (UID: "818028cc-aad0-49bf-bb13-a7eef80f5ca5"). InnerVolumeSpecName "kube-api-access-bxtc7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:47:25.223789 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:25.223707    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxtc7\" (UniqueName: \"kubernetes.io/projected/818028cc-aad0-49bf-bb13-a7eef80f5ca5-kube-api-access-bxtc7\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:25.223789 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:25.223740    5061 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/818028cc-aad0-49bf-bb13-a7eef80f5ca5-home\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:25.223789 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:25.223752    5061 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/818028cc-aad0-49bf-bb13-a7eef80f5ca5-tmp-dir\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:25.223789 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:25.223760    5061 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/818028cc-aad0-49bf-bb13-a7eef80f5ca5-model-cache\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:25.223789 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:25.223768    5061 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/818028cc-aad0-49bf-bb13-a7eef80f5ca5-tls-certs\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:25.223789 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:25.223776    5061 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/818028cc-aad0-49bf-bb13-a7eef80f5ca5-dshm\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:25.754859 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:25.754822    5061 generic.go:356] "Generic (PLEG): container finished" podID="818028cc-aad0-49bf-bb13-a7eef80f5ca5" containerID="4d62194c1741917895d2baeaad40e78cf304bf262326cc3f3914404be734429b" exitCode=0
Oct 30 22:47:25.755288 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:25.754871    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-b1b8bb/simulator-single-main-b1b8bb-kserve-555cb456bf-jwl2j"
Oct 30 22:47:25.755288 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:25.754891    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-b1b8bb/simulator-single-main-b1b8bb-kserve-555cb456bf-jwl2j" event={"ID":"818028cc-aad0-49bf-bb13-a7eef80f5ca5","Type":"ContainerDied","Data":"4d62194c1741917895d2baeaad40e78cf304bf262326cc3f3914404be734429b"}
Oct 30 22:47:25.755288 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:25.754929    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-b1b8bb/simulator-single-main-b1b8bb-kserve-555cb456bf-jwl2j" event={"ID":"818028cc-aad0-49bf-bb13-a7eef80f5ca5","Type":"ContainerDied","Data":"12571cd1d3719eaf92f76c69f3ccec9d9dd52eac4bed544e96115381008a6946"}
Oct 30 22:47:25.755288 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:25.754951    5061 scope.go:122] "RemoveContainer" containerID="4d62194c1741917895d2baeaad40e78cf304bf262326cc3f3914404be734429b"
Oct 30 22:47:25.766839 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:25.766787    5061 scope.go:122] "RemoveContainer" containerID="4d62194c1741917895d2baeaad40e78cf304bf262326cc3f3914404be734429b"
Oct 30 22:47:25.767245 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:47:25.767219    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d62194c1741917895d2baeaad40e78cf304bf262326cc3f3914404be734429b\": container with ID starting with 4d62194c1741917895d2baeaad40e78cf304bf262326cc3f3914404be734429b not found: ID does not exist" containerID="4d62194c1741917895d2baeaad40e78cf304bf262326cc3f3914404be734429b"
Oct 30 22:47:25.767341 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:25.767251    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d62194c1741917895d2baeaad40e78cf304bf262326cc3f3914404be734429b"} err="failed to get container status \"4d62194c1741917895d2baeaad40e78cf304bf262326cc3f3914404be734429b\": rpc error: code = NotFound desc = could not find container \"4d62194c1741917895d2baeaad40e78cf304bf262326cc3f3914404be734429b\": container with ID starting with 4d62194c1741917895d2baeaad40e78cf304bf262326cc3f3914404be734429b not found: ID does not exist"
Oct 30 22:47:25.927333 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:25.927299    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-b1b8bb/simulator-single-main-b1b8bb-kserve-555cb456bf-jwl2j"]
Oct 30 22:47:25.930833 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:25.930802    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-b1b8bb/simulator-single-main-b1b8bb-kserve-555cb456bf-jwl2j"]
Oct 30 22:47:26.126934 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:26.126895    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b1b8bb-596f6c9h5c" podUID="deb10767-aae5-4dd9-a1bd-e2bae05f88ce" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:47:26.127117 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:26.127011    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b1b8bb-7d544ddbclrw4p" podUID="0dab0e3f-d992-4b7f-a958-7887da685867" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:47:26.528798 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:26.528723    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="818028cc-aad0-49bf-bb13-a7eef80f5ca5" path="/var/lib/kubelet/pods/818028cc-aad0-49bf-bb13-a7eef80f5ca5/volumes"
Oct 30 22:47:27.060686 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:27.060659    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-sclrt_5e74071e-0462-42b5-a05c-d90bb638478a/console-operator/2.log"
Oct 30 22:47:27.111198 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:27.111173    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-sclrt_5e74071e-0462-42b5-a05c-d90bb638478a/console-operator/2.log"
Oct 30 22:47:35.668890 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:35.668857    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-5df7564a-kserve-8475d59897-ps2hx"]
Oct 30 22:47:35.669307 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:35.669291    5061 cpu_manager.go:414] "containerMap: removing container" podUID="818028cc-aad0-49bf-bb13-a7eef80f5ca5" containerName="main"
Oct 30 22:47:35.669307 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:35.669302    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="818028cc-aad0-49bf-bb13-a7eef80f5ca5" containerName="main"
Oct 30 22:47:35.669393 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:35.669318    5061 cpu_manager.go:414] "containerMap: removing container" podUID="5733b960-353a-4818-93c6-f1c02943ecb7" containerName="revoke-keys"
Oct 30 22:47:35.669393 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:35.669323    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5733b960-353a-4818-93c6-f1c02943ecb7" containerName="revoke-keys"
Oct 30 22:47:35.669458 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:35.669404    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="818028cc-aad0-49bf-bb13-a7eef80f5ca5" containerName="main"
Oct 30 22:47:35.669458 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:35.669413    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="5733b960-353a-4818-93c6-f1c02943ecb7" containerName="revoke-keys"
Oct 30 22:47:35.672579 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:35.672559    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-5df7564a-kserve-8475d59897-ps2hx"
Oct 30 22:47:35.780714 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:35.780685    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-5df7564a-kserve-self-signed-certs\""
Oct 30 22:47:35.782878 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:35.782856    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-5df7564a-kserve-8475d59897-ps2hx"]
Oct 30 22:47:35.828292 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:35.828260    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/249f677e-8aee-4921-85aa-719ce21f4e98-tmp-dir\") pod \"e2e-same-id-a-5df7564a-kserve-8475d59897-ps2hx\" (UID: \"249f677e-8aee-4921-85aa-719ce21f4e98\") " pod="llm/e2e-same-id-a-5df7564a-kserve-8475d59897-ps2hx"
Oct 30 22:47:35.828494 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:35.828300    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/249f677e-8aee-4921-85aa-719ce21f4e98-tls-certs\") pod \"e2e-same-id-a-5df7564a-kserve-8475d59897-ps2hx\" (UID: \"249f677e-8aee-4921-85aa-719ce21f4e98\") " pod="llm/e2e-same-id-a-5df7564a-kserve-8475d59897-ps2hx"
Oct 30 22:47:35.828494 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:35.828442    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4fz5\" (UniqueName: \"kubernetes.io/projected/249f677e-8aee-4921-85aa-719ce21f4e98-kube-api-access-h4fz5\") pod \"e2e-same-id-a-5df7564a-kserve-8475d59897-ps2hx\" (UID: \"249f677e-8aee-4921-85aa-719ce21f4e98\") " pod="llm/e2e-same-id-a-5df7564a-kserve-8475d59897-ps2hx"
Oct 30 22:47:35.828622 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:35.828491    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/249f677e-8aee-4921-85aa-719ce21f4e98-model-cache\") pod \"e2e-same-id-a-5df7564a-kserve-8475d59897-ps2hx\" (UID: \"249f677e-8aee-4921-85aa-719ce21f4e98\") " pod="llm/e2e-same-id-a-5df7564a-kserve-8475d59897-ps2hx"
Oct 30 22:47:35.828622 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:35.828583    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/249f677e-8aee-4921-85aa-719ce21f4e98-home\") pod \"e2e-same-id-a-5df7564a-kserve-8475d59897-ps2hx\" (UID: \"249f677e-8aee-4921-85aa-719ce21f4e98\") " pod="llm/e2e-same-id-a-5df7564a-kserve-8475d59897-ps2hx"
Oct 30 22:47:35.828622 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:35.828609    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/249f677e-8aee-4921-85aa-719ce21f4e98-dshm\") pod \"e2e-same-id-a-5df7564a-kserve-8475d59897-ps2hx\" (UID: \"249f677e-8aee-4921-85aa-719ce21f4e98\") " pod="llm/e2e-same-id-a-5df7564a-kserve-8475d59897-ps2hx"
Oct 30 22:47:35.929634 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:35.929560    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/249f677e-8aee-4921-85aa-719ce21f4e98-tmp-dir\") pod \"e2e-same-id-a-5df7564a-kserve-8475d59897-ps2hx\" (UID: \"249f677e-8aee-4921-85aa-719ce21f4e98\") " pod="llm/e2e-same-id-a-5df7564a-kserve-8475d59897-ps2hx"
Oct 30 22:47:35.929634 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:35.929594    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/249f677e-8aee-4921-85aa-719ce21f4e98-tls-certs\") pod \"e2e-same-id-a-5df7564a-kserve-8475d59897-ps2hx\" (UID: \"249f677e-8aee-4921-85aa-719ce21f4e98\") " pod="llm/e2e-same-id-a-5df7564a-kserve-8475d59897-ps2hx"
Oct 30 22:47:35.929837 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:35.929659    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4fz5\" (UniqueName: \"kubernetes.io/projected/249f677e-8aee-4921-85aa-719ce21f4e98-kube-api-access-h4fz5\") pod \"e2e-same-id-a-5df7564a-kserve-8475d59897-ps2hx\" (UID: \"249f677e-8aee-4921-85aa-719ce21f4e98\") " pod="llm/e2e-same-id-a-5df7564a-kserve-8475d59897-ps2hx"
Oct 30 22:47:35.929837 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:35.929682    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/249f677e-8aee-4921-85aa-719ce21f4e98-model-cache\") pod \"e2e-same-id-a-5df7564a-kserve-8475d59897-ps2hx\" (UID: \"249f677e-8aee-4921-85aa-719ce21f4e98\") " pod="llm/e2e-same-id-a-5df7564a-kserve-8475d59897-ps2hx"
Oct 30 22:47:35.929837 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:35.929713    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/249f677e-8aee-4921-85aa-719ce21f4e98-home\") pod \"e2e-same-id-a-5df7564a-kserve-8475d59897-ps2hx\" (UID: \"249f677e-8aee-4921-85aa-719ce21f4e98\") " pod="llm/e2e-same-id-a-5df7564a-kserve-8475d59897-ps2hx"
Oct 30 22:47:35.929837 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:35.929734    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/249f677e-8aee-4921-85aa-719ce21f4e98-dshm\") pod \"e2e-same-id-a-5df7564a-kserve-8475d59897-ps2hx\" (UID: \"249f677e-8aee-4921-85aa-719ce21f4e98\") " pod="llm/e2e-same-id-a-5df7564a-kserve-8475d59897-ps2hx"
Oct 30 22:47:35.930032 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:35.930012    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/249f677e-8aee-4921-85aa-719ce21f4e98-model-cache\") pod \"e2e-same-id-a-5df7564a-kserve-8475d59897-ps2hx\" (UID: \"249f677e-8aee-4921-85aa-719ce21f4e98\") " pod="llm/e2e-same-id-a-5df7564a-kserve-8475d59897-ps2hx"
Oct 30 22:47:35.930080 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:35.930031    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/249f677e-8aee-4921-85aa-719ce21f4e98-tmp-dir\") pod \"e2e-same-id-a-5df7564a-kserve-8475d59897-ps2hx\" (UID: \"249f677e-8aee-4921-85aa-719ce21f4e98\") " pod="llm/e2e-same-id-a-5df7564a-kserve-8475d59897-ps2hx"
Oct 30 22:47:35.930128 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:35.930090    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/249f677e-8aee-4921-85aa-719ce21f4e98-home\") pod \"e2e-same-id-a-5df7564a-kserve-8475d59897-ps2hx\" (UID: \"249f677e-8aee-4921-85aa-719ce21f4e98\") " pod="llm/e2e-same-id-a-5df7564a-kserve-8475d59897-ps2hx"
Oct 30 22:47:35.931985 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:35.931960    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/249f677e-8aee-4921-85aa-719ce21f4e98-dshm\") pod \"e2e-same-id-a-5df7564a-kserve-8475d59897-ps2hx\" (UID: \"249f677e-8aee-4921-85aa-719ce21f4e98\") " pod="llm/e2e-same-id-a-5df7564a-kserve-8475d59897-ps2hx"
Oct 30 22:47:35.932273 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:35.932257    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/249f677e-8aee-4921-85aa-719ce21f4e98-tls-certs\") pod \"e2e-same-id-a-5df7564a-kserve-8475d59897-ps2hx\" (UID: \"249f677e-8aee-4921-85aa-719ce21f4e98\") " pod="llm/e2e-same-id-a-5df7564a-kserve-8475d59897-ps2hx"
Oct 30 22:47:35.990117 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:35.990085    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4fz5\" (UniqueName: \"kubernetes.io/projected/249f677e-8aee-4921-85aa-719ce21f4e98-kube-api-access-h4fz5\") pod \"e2e-same-id-a-5df7564a-kserve-8475d59897-ps2hx\" (UID: \"249f677e-8aee-4921-85aa-719ce21f4e98\") " pod="llm/e2e-same-id-a-5df7564a-kserve-8475d59897-ps2hx"
Oct 30 22:47:36.126253 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:36.126213    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b1b8bb-7d544ddbclrw4p" podUID="0dab0e3f-d992-4b7f-a958-7887da685867" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:47:36.126671 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:36.126644    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b1b8bb-596f6c9h5c" podUID="deb10767-aae5-4dd9-a1bd-e2bae05f88ce" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:47:36.284182 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:36.284106    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-5df7564a-kserve-8475d59897-ps2hx"
Oct 30 22:47:36.650473 ip-10-0-130-60 kubenswrapper[5061]: W1030 22:47:36.650436    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod249f677e_8aee_4921_85aa_719ce21f4e98.slice/crio-4437d071162cfef8a5046226c50860978c602b01efd458e1c2c75740a5b3e6f7 WatchSource:0}: Error finding container 4437d071162cfef8a5046226c50860978c602b01efd458e1c2c75740a5b3e6f7: Status 404 returned error can't find the container with id 4437d071162cfef8a5046226c50860978c602b01efd458e1c2c75740a5b3e6f7
Oct 30 22:47:36.654707 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:36.654681    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-5df7564a-kserve-8475d59897-ps2hx"]
Oct 30 22:47:36.820744 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:36.820702    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-5df7564a-kserve-8475d59897-ps2hx" event={"ID":"249f677e-8aee-4921-85aa-719ce21f4e98","Type":"ContainerStarted","Data":"4104fd8194235a0dee65b7ac439792074ab52cc1ae2fc9771c9cde3bd4af0382"}
Oct 30 22:47:36.821394 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:36.821136    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-5df7564a-kserve-8475d59897-ps2hx" event={"ID":"249f677e-8aee-4921-85aa-719ce21f4e98","Type":"ContainerStarted","Data":"4437d071162cfef8a5046226c50860978c602b01efd458e1c2c75740a5b3e6f7"}
Oct 30 22:47:36.942013 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:36.941892    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-5df7564a-kserve-8475d59897-ps2hx" podStartSLOduration=1.9418748940000001 podStartE2EDuration="1.941874894s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:47:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:47:36.939719262 +0000 UTC m=+3910.552489022" watchObservedRunningTime="2026-10-30 22:47:36.941874894 +0000 UTC m=+3910.554644651"
Oct 30 22:47:37.284708 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:37.284624    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-5df7564a-kserve-8475d59897-ps2hx"
Oct 30 22:47:41.887970 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:41.887729    5061 generic.go:356] "Generic (PLEG): container finished" podID="0dab0e3f-d992-4b7f-a958-7887da685867" containerID="c7a426e6760851e021ce2c121ff308a654e2b77687786d90489b6fafa2a2db28" exitCode=0
Oct 30 22:47:41.888892 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:41.887872    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b1b8bb-7d544ddbclrw4p" event={"ID":"0dab0e3f-d992-4b7f-a958-7887da685867","Type":"ContainerDied","Data":"c7a426e6760851e021ce2c121ff308a654e2b77687786d90489b6fafa2a2db28"}
Oct 30 22:47:41.890044 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:41.890017    5061 generic.go:356] "Generic (PLEG): container finished" podID="deb10767-aae5-4dd9-a1bd-e2bae05f88ce" containerID="4b929a7f90467335ed01bdec4a0baa3b750d919aad8ae36c62707a09d9f15555" exitCode=0
Oct 30 22:47:41.890163 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:41.890124    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b1b8bb-596f6c9h5c" event={"ID":"deb10767-aae5-4dd9-a1bd-e2bae05f88ce","Type":"ContainerDied","Data":"4b929a7f90467335ed01bdec4a0baa3b750d919aad8ae36c62707a09d9f15555"}
Oct 30 22:47:42.868728 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:42.868694    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b1b8bb-7d544ddbclrw4p"
Oct 30 22:47:42.962824 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:42.962704    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b1b8bb-596f6c9h5c"
Oct 30 22:47:42.964990 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:42.964015    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0dab0e3f-d992-4b7f-a958-7887da685867-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0dab0e3f-d992-4b7f-a958-7887da685867-plugins-config-volume\") pod \"0dab0e3f-d992-4b7f-a958-7887da685867\" (UID: \"0dab0e3f-d992-4b7f-a958-7887da685867\") "
Oct 30 22:47:42.964990 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:42.964067    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/deb10767-aae5-4dd9-a1bd-e2bae05f88ce-kube-api-access-4b9xg\" (UniqueName: \"kubernetes.io/projected/deb10767-aae5-4dd9-a1bd-e2bae05f88ce-kube-api-access-4b9xg\") pod \"deb10767-aae5-4dd9-a1bd-e2bae05f88ce\" (UID: \"deb10767-aae5-4dd9-a1bd-e2bae05f88ce\") "
Oct 30 22:47:42.964990 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:42.964193    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/deb10767-aae5-4dd9-a1bd-e2bae05f88ce-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/deb10767-aae5-4dd9-a1bd-e2bae05f88ce-plugins-config-volume\") pod \"deb10767-aae5-4dd9-a1bd-e2bae05f88ce\" (UID: \"deb10767-aae5-4dd9-a1bd-e2bae05f88ce\") "
Oct 30 22:47:42.964990 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:42.964256    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0dab0e3f-d992-4b7f-a958-7887da685867-kube-api-access-5q4m2\" (UniqueName: \"kubernetes.io/projected/0dab0e3f-d992-4b7f-a958-7887da685867-kube-api-access-5q4m2\") pod \"0dab0e3f-d992-4b7f-a958-7887da685867\" (UID: \"0dab0e3f-d992-4b7f-a958-7887da685867\") "
Oct 30 22:47:42.967354 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:42.967301    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0dab0e3f-d992-4b7f-a958-7887da685867-plugins-config-volume" pod "0dab0e3f-d992-4b7f-a958-7887da685867" (UID: "0dab0e3f-d992-4b7f-a958-7887da685867"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:47:42.967759 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:42.967732    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/deb10767-aae5-4dd9-a1bd-e2bae05f88ce-plugins-config-volume" pod "deb10767-aae5-4dd9-a1bd-e2bae05f88ce" (UID: "deb10767-aae5-4dd9-a1bd-e2bae05f88ce"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:47:42.970123 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:42.970100    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b1b8bb-7d544ddbclrw4p"
Oct 30 22:47:42.995184 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:42.981570    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b1b8bb-7d544ddbclrw4p" event={"ID":"0dab0e3f-d992-4b7f-a958-7887da685867","Type":"ContainerDied","Data":"5a86b705a6890e4a2f40252e2f4e0972610bc627f8d00558426ac8f94d782a40"}
Oct 30 22:47:43.012180 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:43.012133    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b1b8bb-596f6c9h5c" event={"ID":"deb10767-aae5-4dd9-a1bd-e2bae05f88ce","Type":"ContainerDied","Data":"b58d793fd1df5b713a0314f042f7a2f97206b448081022ab9b5f2c6f23226564"}
Oct 30 22:47:43.012404 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:42.995824    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/deb10767-aae5-4dd9-a1bd-e2bae05f88ce-kube-api-access-4b9xg" pod "deb10767-aae5-4dd9-a1bd-e2bae05f88ce" (UID: "deb10767-aae5-4dd9-a1bd-e2bae05f88ce"). InnerVolumeSpecName "kube-api-access-4b9xg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:47:43.012519 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:42.996353    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dab0e3f-d992-4b7f-a958-7887da685867-kube-api-access-5q4m2" pod "0dab0e3f-d992-4b7f-a958-7887da685867" (UID: "0dab0e3f-d992-4b7f-a958-7887da685867"). InnerVolumeSpecName "kube-api-access-5q4m2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:47:43.012652 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:42.991181    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b1b8bb-596f6c9h5c"
Oct 30 22:47:43.016297 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:43.016270    5061 scope.go:122] "RemoveContainer" containerID="c7a426e6760851e021ce2c121ff308a654e2b77687786d90489b6fafa2a2db28"
Oct 30 22:47:43.069583 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:43.066436    5061 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0dab0e3f-d992-4b7f-a958-7887da685867-plugins-config-volume\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:43.069583 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:43.066473    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4b9xg\" (UniqueName: \"kubernetes.io/projected/deb10767-aae5-4dd9-a1bd-e2bae05f88ce-kube-api-access-4b9xg\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:43.069583 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:43.066489    5061 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/deb10767-aae5-4dd9-a1bd-e2bae05f88ce-plugins-config-volume\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:43.069583 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:43.066503    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5q4m2\" (UniqueName: \"kubernetes.io/projected/0dab0e3f-d992-4b7f-a958-7887da685867-kube-api-access-5q4m2\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:43.242849 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:43.218516    5061 scope.go:122] "RemoveContainer" containerID="4b929a7f90467335ed01bdec4a0baa3b750d919aad8ae36c62707a09d9f15555"
Oct 30 22:47:43.499631 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:43.499363    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-b1b8bb-7d544ddbclrw4p"]
Oct 30 22:47:43.511815 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:43.511785    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-b1b8bb-7d544ddbclrw4p"]
Oct 30 22:47:43.684289 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:43.651462    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-b1b8bb-596f6c9h5c"]
Oct 30 22:47:43.700463 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:43.700421    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-b1b8bb-596f6c9h5c"]
Oct 30 22:47:44.527636 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:44.527603    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0dab0e3f-d992-4b7f-a958-7887da685867" path="/var/lib/kubelet/pods/0dab0e3f-d992-4b7f-a958-7887da685867/volumes"
Oct 30 22:47:44.528031 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:44.527996    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="deb10767-aae5-4dd9-a1bd-e2bae05f88ce" path="/var/lib/kubelet/pods/deb10767-aae5-4dd9-a1bd-e2bae05f88ce/volumes"
Oct 30 22:47:46.284929 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:46.284885    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-5df7564a-kserve-8475d59897-ps2hx"
Oct 30 22:47:46.298139 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:46.298103    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-5df7564a-kserve-8475d59897-ps2hx"
Oct 30 22:47:47.024379 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:47:47.024347    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-5df7564a-kserve-8475d59897-ps2hx"
Oct 30 22:47:54.665417 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:47:54.665376    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 22:48:24.687217 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:48:24.687095    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 22:48:26.545753 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:48:26.545708    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-nbmc8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:48:26.546402 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:48:26.546350    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-nbmc8" podUID="a77dd560-9aff-4292-b430-fa9c72d8c053" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:48:29.982437 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:48:29.982405    5061 scope.go:122] "RemoveContainer" containerID="19b70de5518761d3a785e08ad0b74c7dbd032473d6907a5303df296e68e4c342"
Oct 30 22:48:29.992684 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:48:29.992665    5061 scope.go:122] "RemoveContainer" containerID="3af2b05941219699ab795d80bacdde95e4e483c670363555929996a556377e09"
Oct 30 22:48:54.682429 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:48:54.682387    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 22:48:56.548056 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:48:56.547620    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-nbmc8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:48:56.548056 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:48:56.547676    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-nbmc8" podUID="a77dd560-9aff-4292-b430-fa9c72d8c053" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:49:04.868805 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:49:04.868769    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-5df7564a-kserve-8475d59897-ps2hx"]
Oct 30 22:49:11.559821 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:49:11.559724    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-nbmc8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:49:11.572550 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:49:11.566593    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-nbmc8" podUID="a77dd560-9aff-4292-b430-fa9c72d8c053" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:49:19.891553 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:49:19.890707    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-5df7564a-kserve-8475d59897-ps2hx" podUID="249f677e-8aee-4921-85aa-719ce21f4e98" containerName="main" containerID="cri-o://4104fd8194235a0dee65b7ac439792074ab52cc1ae2fc9771c9cde3bd4af0382" gracePeriod=45
Oct 30 22:49:20.028551 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:49:20.022701    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-same-id-a-5df7564a-kserve-8475d59897-ps2hx" podUID="249f677e-8aee-4921-85aa-719ce21f4e98" containerName="main" probeResult="failure" output="Get \"https://10.132.0.226:8000/ready\": dial tcp 10.132.0.226:8000: connect: connection refused"
Oct 30 22:49:20.404494 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:49:20.404467    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-5df7564a-kserve-8475d59897-ps2hx"
Oct 30 22:49:20.469625 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:49:20.469592    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/249f677e-8aee-4921-85aa-719ce21f4e98-dshm\" (UniqueName: \"kubernetes.io/empty-dir/249f677e-8aee-4921-85aa-719ce21f4e98-dshm\") pod \"249f677e-8aee-4921-85aa-719ce21f4e98\" (UID: \"249f677e-8aee-4921-85aa-719ce21f4e98\") "
Oct 30 22:49:20.469787 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:49:20.469633    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/249f677e-8aee-4921-85aa-719ce21f4e98-home\" (UniqueName: \"kubernetes.io/empty-dir/249f677e-8aee-4921-85aa-719ce21f4e98-home\") pod \"249f677e-8aee-4921-85aa-719ce21f4e98\" (UID: \"249f677e-8aee-4921-85aa-719ce21f4e98\") "
Oct 30 22:49:20.469787 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:49:20.469667    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/249f677e-8aee-4921-85aa-719ce21f4e98-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/249f677e-8aee-4921-85aa-719ce21f4e98-tmp-dir\") pod \"249f677e-8aee-4921-85aa-719ce21f4e98\" (UID: \"249f677e-8aee-4921-85aa-719ce21f4e98\") "
Oct 30 22:49:20.469787 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:49:20.469718    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/249f677e-8aee-4921-85aa-719ce21f4e98-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/249f677e-8aee-4921-85aa-719ce21f4e98-model-cache\") pod \"249f677e-8aee-4921-85aa-719ce21f4e98\" (UID: \"249f677e-8aee-4921-85aa-719ce21f4e98\") "
Oct 30 22:49:20.469787 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:49:20.469760    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/249f677e-8aee-4921-85aa-719ce21f4e98-kube-api-access-h4fz5\" (UniqueName: \"kubernetes.io/projected/249f677e-8aee-4921-85aa-719ce21f4e98-kube-api-access-h4fz5\") pod \"249f677e-8aee-4921-85aa-719ce21f4e98\" (UID: \"249f677e-8aee-4921-85aa-719ce21f4e98\") "
Oct 30 22:49:20.470039 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:49:20.469917    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/249f677e-8aee-4921-85aa-719ce21f4e98-home" pod "249f677e-8aee-4921-85aa-719ce21f4e98" (UID: "249f677e-8aee-4921-85aa-719ce21f4e98"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:49:20.470039 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:49:20.469947    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/249f677e-8aee-4921-85aa-719ce21f4e98-tls-certs\" (UniqueName: \"kubernetes.io/secret/249f677e-8aee-4921-85aa-719ce21f4e98-tls-certs\") pod \"249f677e-8aee-4921-85aa-719ce21f4e98\" (UID: \"249f677e-8aee-4921-85aa-719ce21f4e98\") "
Oct 30 22:49:20.470039 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:49:20.469954    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/249f677e-8aee-4921-85aa-719ce21f4e98-tmp-dir" pod "249f677e-8aee-4921-85aa-719ce21f4e98" (UID: "249f677e-8aee-4921-85aa-719ce21f4e98"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:49:20.470039 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:49:20.470023    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/249f677e-8aee-4921-85aa-719ce21f4e98-model-cache" pod "249f677e-8aee-4921-85aa-719ce21f4e98" (UID: "249f677e-8aee-4921-85aa-719ce21f4e98"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:49:20.470306 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:49:20.470290    5061 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/249f677e-8aee-4921-85aa-719ce21f4e98-model-cache\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:49:20.470388 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:49:20.470307    5061 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/249f677e-8aee-4921-85aa-719ce21f4e98-home\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:49:20.470388 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:49:20.470316    5061 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/249f677e-8aee-4921-85aa-719ce21f4e98-tmp-dir\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:49:20.471931 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:49:20.471905    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/249f677e-8aee-4921-85aa-719ce21f4e98-dshm" pod "249f677e-8aee-4921-85aa-719ce21f4e98" (UID: "249f677e-8aee-4921-85aa-719ce21f4e98"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:49:20.472036 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:49:20.472019    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/249f677e-8aee-4921-85aa-719ce21f4e98-kube-api-access-h4fz5" pod "249f677e-8aee-4921-85aa-719ce21f4e98" (UID: "249f677e-8aee-4921-85aa-719ce21f4e98"). InnerVolumeSpecName "kube-api-access-h4fz5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:49:20.472078 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:49:20.472061    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/249f677e-8aee-4921-85aa-719ce21f4e98-tls-certs" pod "249f677e-8aee-4921-85aa-719ce21f4e98" (UID: "249f677e-8aee-4921-85aa-719ce21f4e98"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:49:20.570899 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:49:20.570870    5061 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/249f677e-8aee-4921-85aa-719ce21f4e98-tls-certs\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:49:20.570899 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:49:20.570898    5061 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/249f677e-8aee-4921-85aa-719ce21f4e98-dshm\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:49:20.571066 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:49:20.570911    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4fz5\" (UniqueName: \"kubernetes.io/projected/249f677e-8aee-4921-85aa-719ce21f4e98-kube-api-access-h4fz5\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:49:20.700063 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:49:20.700027    5061 generic.go:356] "Generic (PLEG): container finished" podID="249f677e-8aee-4921-85aa-719ce21f4e98" containerID="4104fd8194235a0dee65b7ac439792074ab52cc1ae2fc9771c9cde3bd4af0382" exitCode=0
Oct 30 22:49:20.700252 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:49:20.700086    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-5df7564a-kserve-8475d59897-ps2hx"
Oct 30 22:49:20.700252 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:49:20.700117    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-5df7564a-kserve-8475d59897-ps2hx" event={"ID":"249f677e-8aee-4921-85aa-719ce21f4e98","Type":"ContainerDied","Data":"4104fd8194235a0dee65b7ac439792074ab52cc1ae2fc9771c9cde3bd4af0382"}
Oct 30 22:49:20.700252 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:49:20.700170    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-5df7564a-kserve-8475d59897-ps2hx" event={"ID":"249f677e-8aee-4921-85aa-719ce21f4e98","Type":"ContainerDied","Data":"4437d071162cfef8a5046226c50860978c602b01efd458e1c2c75740a5b3e6f7"}
Oct 30 22:49:20.700252 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:49:20.700193    5061 scope.go:122] "RemoveContainer" containerID="4104fd8194235a0dee65b7ac439792074ab52cc1ae2fc9771c9cde3bd4af0382"
Oct 30 22:49:20.709075 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:49:20.709052    5061 scope.go:122] "RemoveContainer" containerID="4104fd8194235a0dee65b7ac439792074ab52cc1ae2fc9771c9cde3bd4af0382"
Oct 30 22:49:20.709458 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:49:20.709423    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4104fd8194235a0dee65b7ac439792074ab52cc1ae2fc9771c9cde3bd4af0382\": container with ID starting with 4104fd8194235a0dee65b7ac439792074ab52cc1ae2fc9771c9cde3bd4af0382 not found: ID does not exist" containerID="4104fd8194235a0dee65b7ac439792074ab52cc1ae2fc9771c9cde3bd4af0382"
Oct 30 22:49:20.709540 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:49:20.709474    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4104fd8194235a0dee65b7ac439792074ab52cc1ae2fc9771c9cde3bd4af0382"} err="failed to get container status \"4104fd8194235a0dee65b7ac439792074ab52cc1ae2fc9771c9cde3bd4af0382\": rpc error: code = NotFound desc = could not find container \"4104fd8194235a0dee65b7ac439792074ab52cc1ae2fc9771c9cde3bd4af0382\": container with ID starting with 4104fd8194235a0dee65b7ac439792074ab52cc1ae2fc9771c9cde3bd4af0382 not found: ID does not exist"
Oct 30 22:49:20.872733 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:49:20.872706    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-5df7564a-kserve-8475d59897-ps2hx"]
Oct 30 22:49:20.877945 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:49:20.877919    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-5df7564a-kserve-8475d59897-ps2hx"]
Oct 30 22:49:22.527539 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:49:22.527502    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="249f677e-8aee-4921-85aa-719ce21f4e98" path="/var/lib/kubelet/pods/249f677e-8aee-4921-85aa-719ce21f4e98/volumes"
Oct 30 22:49:24.666307 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:49:24.666269    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 22:49:36.489552 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:49:36.487632    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-8t2sw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:49:36.489552 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:49:36.487714    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8t2sw" podUID="d079dbb3-5d17-4dc9-9cf2-f07629b9f1f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:49:54.665893 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:49:54.665854    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 22:49:55.275122 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:49:55.275078    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5fd44bb67c-4fxkc"]
Oct 30 22:49:55.275616 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:49:55.275593    5061 cpu_manager.go:414] "containerMap: removing container" podUID="deb10767-aae5-4dd9-a1bd-e2bae05f88ce" containerName="payload-pre-processing"
Oct 30 22:49:55.275616 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:49:55.275614    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="deb10767-aae5-4dd9-a1bd-e2bae05f88ce" containerName="payload-pre-processing"
Oct 30 22:49:55.275799 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:49:55.275642    5061 cpu_manager.go:414] "containerMap: removing container" podUID="0dab0e3f-d992-4b7f-a958-7887da685867" containerName="payload-processing"
Oct 30 22:49:55.275799 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:49:55.275648    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0dab0e3f-d992-4b7f-a958-7887da685867" containerName="payload-processing"
Oct 30 22:49:55.275799 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:49:55.275664    5061 cpu_manager.go:414] "containerMap: removing container" podUID="249f677e-8aee-4921-85aa-719ce21f4e98" containerName="main"
Oct 30 22:49:55.275799 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:49:55.275673    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="249f677e-8aee-4921-85aa-719ce21f4e98" containerName="main"
Oct 30 22:49:55.275799 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:49:55.275778    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="deb10767-aae5-4dd9-a1bd-e2bae05f88ce" containerName="payload-pre-processing"
Oct 30 22:49:55.275799 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:49:55.275793    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="0dab0e3f-d992-4b7f-a958-7887da685867" containerName="payload-processing"
Oct 30 22:49:55.276125 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:49:55.275805    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="249f677e-8aee-4921-85aa-719ce21f4e98" containerName="main"
Oct 30 22:49:55.278880 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:49:55.278860    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5fd44bb67c-4fxkc"
Oct 30 22:49:55.287708 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:49:55.287681    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08901302-db0c-4cd0-8ab8-d9d43e0c0c6b-metrics-certs\") pod \"maas-api-5fd44bb67c-4fxkc\" (UID: \"08901302-db0c-4cd0-8ab8-d9d43e0c0c6b\") " pod="odh-ai-gateway-infra/maas-api-5fd44bb67c-4fxkc"
Oct 30 22:49:55.287800 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:49:55.287739    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kf5g9\" (UniqueName: \"kubernetes.io/projected/08901302-db0c-4cd0-8ab8-d9d43e0c0c6b-kube-api-access-kf5g9\") pod \"maas-api-5fd44bb67c-4fxkc\" (UID: \"08901302-db0c-4cd0-8ab8-d9d43e0c0c6b\") " pod="odh-ai-gateway-infra/maas-api-5fd44bb67c-4fxkc"
Oct 30 22:49:55.287858 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:49:55.287803    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08901302-db0c-4cd0-8ab8-d9d43e0c0c6b-maas-api-tls\") pod \"maas-api-5fd44bb67c-4fxkc\" (UID: \"08901302-db0c-4cd0-8ab8-d9d43e0c0c6b\") " pod="odh-ai-gateway-infra/maas-api-5fd44bb67c-4fxkc"
Oct 30 22:49:55.287957 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:49:55.287861    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08901302-db0c-4cd0-8ab8-d9d43e0c0c6b-openshift-service-ca\") pod \"maas-api-5fd44bb67c-4fxkc\" (UID: \"08901302-db0c-4cd0-8ab8-d9d43e0c0c6b\") " pod="odh-ai-gateway-infra/maas-api-5fd44bb67c-4fxkc"
Oct 30 22:49:55.333658 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:49:55.333620    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:49:55.333821 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:49:55.333685    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Oct 30 22:49:55.386456 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:49:55.386417    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:49:55.386456 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:49:55.386417    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-rsvrw\""
Oct 30 22:49:55.386750 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:49:55.386576    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 30 22:49:55.388992 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:49:55.388965    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5fd44bb67c-4fxkc"]
Oct 30 22:49:55.389158 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:49:55.389137    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08901302-db0c-4cd0-8ab8-d9d43e0c0c6b-maas-api-tls\") pod \"maas-api-5fd44bb67c-4fxkc\" (UID: \"08901302-db0c-4cd0-8ab8-d9d43e0c0c6b\") " pod="odh-ai-gateway-infra/maas-api-5fd44bb67c-4fxkc"
Oct 30 22:49:55.389233 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:49:55.389203    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08901302-db0c-4cd0-8ab8-d9d43e0c0c6b-openshift-service-ca\") pod \"maas-api-5fd44bb67c-4fxkc\" (UID: \"08901302-db0c-4cd0-8ab8-d9d43e0c0c6b\") " pod="odh-ai-gateway-infra/maas-api-5fd44bb67c-4fxkc"
Oct 30 22:49:55.389305 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:49:55.389290    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08901302-db0c-4cd0-8ab8-d9d43e0c0c6b-metrics-certs\") pod \"maas-api-5fd44bb67c-4fxkc\" (UID: \"08901302-db0c-4cd0-8ab8-d9d43e0c0c6b\") " pod="odh-ai-gateway-infra/maas-api-5fd44bb67c-4fxkc"
Oct 30 22:49:55.389368 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:49:55.389332    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kf5g9\" (UniqueName: \"kubernetes.io/projected/08901302-db0c-4cd0-8ab8-d9d43e0c0c6b-kube-api-access-kf5g9\") pod \"maas-api-5fd44bb67c-4fxkc\" (UID: \"08901302-db0c-4cd0-8ab8-d9d43e0c0c6b\") " pod="odh-ai-gateway-infra/maas-api-5fd44bb67c-4fxkc"
Oct 30 22:49:55.389927 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:49:55.389904    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08901302-db0c-4cd0-8ab8-d9d43e0c0c6b-openshift-service-ca\") pod \"maas-api-5fd44bb67c-4fxkc\" (UID: \"08901302-db0c-4cd0-8ab8-d9d43e0c0c6b\") " pod="odh-ai-gateway-infra/maas-api-5fd44bb67c-4fxkc"
Oct 30 22:49:55.391896 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:49:55.391869    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08901302-db0c-4cd0-8ab8-d9d43e0c0c6b-maas-api-tls\") pod \"maas-api-5fd44bb67c-4fxkc\" (UID: \"08901302-db0c-4cd0-8ab8-d9d43e0c0c6b\") " pod="odh-ai-gateway-infra/maas-api-5fd44bb67c-4fxkc"
Oct 30 22:49:55.392008 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:49:55.391934    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08901302-db0c-4cd0-8ab8-d9d43e0c0c6b-metrics-certs\") pod \"maas-api-5fd44bb67c-4fxkc\" (UID: \"08901302-db0c-4cd0-8ab8-d9d43e0c0c6b\") " pod="odh-ai-gateway-infra/maas-api-5fd44bb67c-4fxkc"
Oct 30 22:49:55.449170 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:49:55.449145    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kf5g9\" (UniqueName: \"kubernetes.io/projected/08901302-db0c-4cd0-8ab8-d9d43e0c0c6b-kube-api-access-kf5g9\") pod \"maas-api-5fd44bb67c-4fxkc\" (UID: \"08901302-db0c-4cd0-8ab8-d9d43e0c0c6b\") " pod="odh-ai-gateway-infra/maas-api-5fd44bb67c-4fxkc"
Oct 30 22:49:55.594078 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:49:55.594054    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5fd44bb67c-4fxkc"
Oct 30 22:49:55.959865 ip-10-0-130-60 kubenswrapper[5061]: W1030 22:49:55.959826    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08901302_db0c_4cd0_8ab8_d9d43e0c0c6b.slice/crio-3ca43cea2b056018319c4886ed2eca2f7ec439814c0f4ce6d661a9e2d24d8d45 WatchSource:0}: Error finding container 3ca43cea2b056018319c4886ed2eca2f7ec439814c0f4ce6d661a9e2d24d8d45: Status 404 returned error can't find the container with id 3ca43cea2b056018319c4886ed2eca2f7ec439814c0f4ce6d661a9e2d24d8d45
Oct 30 22:49:55.961446 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:49:55.961425    5061 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 22:49:55.963296 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:49:55.963271    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5fd44bb67c-4fxkc"]
Oct 30 22:49:56.919674 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:49:56.919642    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5fd44bb67c-4fxkc" event={"ID":"08901302-db0c-4cd0-8ab8-d9d43e0c0c6b","Type":"ContainerStarted","Data":"9d653ed5809ee2e9233f04ff59e8ae43594a0b693012108d848b80a82b1ce137"}
Oct 30 22:49:56.919842 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:49:56.919700    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5fd44bb67c-4fxkc"
Oct 30 22:49:56.919842 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:49:56.919717    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5fd44bb67c-4fxkc" event={"ID":"08901302-db0c-4cd0-8ab8-d9d43e0c0c6b","Type":"ContainerStarted","Data":"3ca43cea2b056018319c4886ed2eca2f7ec439814c0f4ce6d661a9e2d24d8d45"}
Oct 30 22:49:57.042192 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:49:57.042141    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5fd44bb67c-4fxkc" podStartSLOduration=1.183986849 podStartE2EDuration="2.042127881s" totalImagesPullingTime="858.141032ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:49:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:49:57.040715825 +0000 UTC m=+4050.653485582" watchObservedRunningTime="2026-10-30 22:49:57.042127881 +0000 UTC m=+4050.654897639"
Oct 30 22:50:02.931735 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:50:02.931700    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5fd44bb67c-4fxkc"
Oct 30 22:50:09.567379 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:50:09.567343    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8756f17a-5f7f9db77d-hlc8c"]
Oct 30 22:50:09.570837 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:50:09.570820    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8756f17a-5f7f9db77d-hlc8c"
Oct 30 22:50:09.603600 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:50:09.603567    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f7cbb3b-d4d4-4523-a596-52c9fe90aad0-metrics-certs\") pod \"maas-api-e2e-ait-8756f17a-5f7f9db77d-hlc8c\" (UID: \"7f7cbb3b-d4d4-4523-a596-52c9fe90aad0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8756f17a-5f7f9db77d-hlc8c"
Oct 30 22:50:09.603760 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:50:09.603614    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f7cbb3b-d4d4-4523-a596-52c9fe90aad0-openshift-service-ca\") pod \"maas-api-e2e-ait-8756f17a-5f7f9db77d-hlc8c\" (UID: \"7f7cbb3b-d4d4-4523-a596-52c9fe90aad0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8756f17a-5f7f9db77d-hlc8c"
Oct 30 22:50:09.603760 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:50:09.603728    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f7cbb3b-d4d4-4523-a596-52c9fe90aad0-maas-api-tls\") pod \"maas-api-e2e-ait-8756f17a-5f7f9db77d-hlc8c\" (UID: \"7f7cbb3b-d4d4-4523-a596-52c9fe90aad0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8756f17a-5f7f9db77d-hlc8c"
Oct 30 22:50:09.603760 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:50:09.603756    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrtjg\" (UniqueName: \"kubernetes.io/projected/7f7cbb3b-d4d4-4523-a596-52c9fe90aad0-kube-api-access-xrtjg\") pod \"maas-api-e2e-ait-8756f17a-5f7f9db77d-hlc8c\" (UID: \"7f7cbb3b-d4d4-4523-a596-52c9fe90aad0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8756f17a-5f7f9db77d-hlc8c"
Oct 30 22:50:09.630964 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:50:09.630931    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8756f17a-5f7f9db77d-hlc8c"]
Oct 30 22:50:09.681385 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:50:09.681350    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-8756f17a-serving-cert\""
Oct 30 22:50:09.704508 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:50:09.704476    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f7cbb3b-d4d4-4523-a596-52c9fe90aad0-openshift-service-ca\") pod \"maas-api-e2e-ait-8756f17a-5f7f9db77d-hlc8c\" (UID: \"7f7cbb3b-d4d4-4523-a596-52c9fe90aad0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8756f17a-5f7f9db77d-hlc8c"
Oct 30 22:50:09.704716 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:50:09.704558    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f7cbb3b-d4d4-4523-a596-52c9fe90aad0-maas-api-tls\") pod \"maas-api-e2e-ait-8756f17a-5f7f9db77d-hlc8c\" (UID: \"7f7cbb3b-d4d4-4523-a596-52c9fe90aad0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8756f17a-5f7f9db77d-hlc8c"
Oct 30 22:50:09.704716 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:50:09.704581    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrtjg\" (UniqueName: \"kubernetes.io/projected/7f7cbb3b-d4d4-4523-a596-52c9fe90aad0-kube-api-access-xrtjg\") pod \"maas-api-e2e-ait-8756f17a-5f7f9db77d-hlc8c\" (UID: \"7f7cbb3b-d4d4-4523-a596-52c9fe90aad0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8756f17a-5f7f9db77d-hlc8c"
Oct 30 22:50:09.704716 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:50:09.704661    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f7cbb3b-d4d4-4523-a596-52c9fe90aad0-metrics-certs\") pod \"maas-api-e2e-ait-8756f17a-5f7f9db77d-hlc8c\" (UID: \"7f7cbb3b-d4d4-4523-a596-52c9fe90aad0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8756f17a-5f7f9db77d-hlc8c"
Oct 30 22:50:09.705115 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:50:09.705086    5061 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-8756f17a-serving-cert: secret "maas-api-e2e-ait-8756f17a-serving-cert" not found
Oct 30 22:50:09.705265 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:50:09.705158    5061 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f7cbb3b-d4d4-4523-a596-52c9fe90aad0-maas-api-tls podName:7f7cbb3b-d4d4-4523-a596-52c9fe90aad0 nodeName:}" failed. No retries permitted until 2026-10-30 22:50:10.205137325 +0000 UTC m=+4063.817907070 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7f7cbb3b-d4d4-4523-a596-52c9fe90aad0-maas-api-tls") pod "maas-api-e2e-ait-8756f17a-5f7f9db77d-hlc8c" (UID: "7f7cbb3b-d4d4-4523-a596-52c9fe90aad0") : secret "maas-api-e2e-ait-8756f17a-serving-cert" not found
Oct 30 22:50:09.705344 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:50:09.705271    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f7cbb3b-d4d4-4523-a596-52c9fe90aad0-openshift-service-ca\") pod \"maas-api-e2e-ait-8756f17a-5f7f9db77d-hlc8c\" (UID: \"7f7cbb3b-d4d4-4523-a596-52c9fe90aad0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8756f17a-5f7f9db77d-hlc8c"
Oct 30 22:50:09.707712 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:50:09.707685    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f7cbb3b-d4d4-4523-a596-52c9fe90aad0-metrics-certs\") pod \"maas-api-e2e-ait-8756f17a-5f7f9db77d-hlc8c\" (UID: \"7f7cbb3b-d4d4-4523-a596-52c9fe90aad0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8756f17a-5f7f9db77d-hlc8c"
Oct 30 22:50:09.766078 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:50:09.766050    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrtjg\" (UniqueName: \"kubernetes.io/projected/7f7cbb3b-d4d4-4523-a596-52c9fe90aad0-kube-api-access-xrtjg\") pod \"maas-api-e2e-ait-8756f17a-5f7f9db77d-hlc8c\" (UID: \"7f7cbb3b-d4d4-4523-a596-52c9fe90aad0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8756f17a-5f7f9db77d-hlc8c"
Oct 30 22:50:10.212821 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:50:10.212759    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f7cbb3b-d4d4-4523-a596-52c9fe90aad0-maas-api-tls\") pod \"maas-api-e2e-ait-8756f17a-5f7f9db77d-hlc8c\" (UID: \"7f7cbb3b-d4d4-4523-a596-52c9fe90aad0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8756f17a-5f7f9db77d-hlc8c"
Oct 30 22:50:10.233368 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:50:10.232406    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f7cbb3b-d4d4-4523-a596-52c9fe90aad0-maas-api-tls\") pod \"maas-api-e2e-ait-8756f17a-5f7f9db77d-hlc8c\" (UID: \"7f7cbb3b-d4d4-4523-a596-52c9fe90aad0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8756f17a-5f7f9db77d-hlc8c"
Oct 30 22:50:10.316988 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:50:10.316950    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8756f17a-7cddb85964-lnvkn"]
Oct 30 22:50:10.320770 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:50:10.320751    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8756f17a-7cddb85964-lnvkn"
Oct 30 22:50:10.377814 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:50:10.377773    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-8756f17a-dockercfg-fjwx4\""
Oct 30 22:50:10.377984 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:50:10.377835    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-8756f17a\""
Oct 30 22:50:10.379237 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:50:10.379211    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8756f17a-7cddb85964-lnvkn"]
Oct 30 22:50:10.416749 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:50:10.416710    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbmgx\" (UniqueName: \"kubernetes.io/projected/1da7d0be-1c41-4da5-b4f4-093295e92841-kube-api-access-cbmgx\") pod \"payload-processing-e2e-ait-8756f17a-7cddb85964-lnvkn\" (UID: \"1da7d0be-1c41-4da5-b4f4-093295e92841\") " pod="openshift-ingress/payload-processing-e2e-ait-8756f17a-7cddb85964-lnvkn"
Oct 30 22:50:10.416923 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:50:10.416903    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1da7d0be-1c41-4da5-b4f4-093295e92841-plugins-config-volume\") pod \"payload-processing-e2e-ait-8756f17a-7cddb85964-lnvkn\" (UID: \"1da7d0be-1c41-4da5-b4f4-093295e92841\") " pod="openshift-ingress/payload-processing-e2e-ait-8756f17a-7cddb85964-lnvkn"
Oct 30 22:50:10.483281 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:50:10.483201    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8756f17a-5f7f9db77d-hlc8c"
Oct 30 22:50:10.518195 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:50:10.518164    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbmgx\" (UniqueName: \"kubernetes.io/projected/1da7d0be-1c41-4da5-b4f4-093295e92841-kube-api-access-cbmgx\") pod \"payload-processing-e2e-ait-8756f17a-7cddb85964-lnvkn\" (UID: \"1da7d0be-1c41-4da5-b4f4-093295e92841\") " pod="openshift-ingress/payload-processing-e2e-ait-8756f17a-7cddb85964-lnvkn"
Oct 30 22:50:10.518378 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:50:10.518294    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1da7d0be-1c41-4da5-b4f4-093295e92841-plugins-config-volume\") pod \"payload-processing-e2e-ait-8756f17a-7cddb85964-lnvkn\" (UID: \"1da7d0be-1c41-4da5-b4f4-093295e92841\") " pod="openshift-ingress/payload-processing-e2e-ait-8756f17a-7cddb85964-lnvkn"
Oct 30 22:50:10.518989 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:50:10.518964    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1da7d0be-1c41-4da5-b4f4-093295e92841-plugins-config-volume\") pod \"payload-processing-e2e-ait-8756f17a-7cddb85964-lnvkn\" (UID: \"1da7d0be-1c41-4da5-b4f4-093295e92841\") " pod="openshift-ingress/payload-processing-e2e-ait-8756f17a-7cddb85964-lnvkn"
Oct 30 22:50:10.579117 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:50:10.579091    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbmgx\" (UniqueName: \"kubernetes.io/projected/1da7d0be-1c41-4da5-b4f4-093295e92841-kube-api-access-cbmgx\") pod \"payload-processing-e2e-ait-8756f17a-7cddb85964-lnvkn\" (UID: \"1da7d0be-1c41-4da5-b4f4-093295e92841\") " pod="openshift-ingress/payload-processing-e2e-ait-8756f17a-7cddb85964-lnvkn"
Oct 30 22:50:10.633969 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:50:10.633941    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8756f17a-7cddb85964-lnvkn"
Oct 30 22:50:10.851318 ip-10-0-130-60 kubenswrapper[5061]: W1030 22:50:10.851286    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f7cbb3b_d4d4_4523_a596_52c9fe90aad0.slice/crio-1a82be054653ed2e59e2e8d5bb1c47ebfe6b9a42ea1b8612cd7e82d47f36d499 WatchSource:0}: Error finding container 1a82be054653ed2e59e2e8d5bb1c47ebfe6b9a42ea1b8612cd7e82d47f36d499: Status 404 returned error can't find the container with id 1a82be054653ed2e59e2e8d5bb1c47ebfe6b9a42ea1b8612cd7e82d47f36d499
Oct 30 22:50:10.855018 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:50:10.854991    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8756f17a-5f7f9db77d-hlc8c"]
Oct 30 22:50:10.991668 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:50:10.991631    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8756f17a-5f7f9db77d-hlc8c" event={"ID":"7f7cbb3b-d4d4-4523-a596-52c9fe90aad0","Type":"ContainerStarted","Data":"1a82be054653ed2e59e2e8d5bb1c47ebfe6b9a42ea1b8612cd7e82d47f36d499"}
Oct 30 22:50:11.001271 ip-10-0-130-60 kubenswrapper[5061]: W1030 22:50:11.001236    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1da7d0be_1c41_4da5_b4f4_093295e92841.slice/crio-b988de2324da871f8c5739687be4920509697aa1c6b5a57030abccaf9ddb13de WatchSource:0}: Error finding container b988de2324da871f8c5739687be4920509697aa1c6b5a57030abccaf9ddb13de: Status 404 returned error can't find the container with id b988de2324da871f8c5739687be4920509697aa1c6b5a57030abccaf9ddb13de
Oct 30 22:50:11.005196 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:50:11.005172    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8756f17a-7cddb85964-lnvkn"]
Oct 30 22:50:11.997763 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:50:11.997729    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8756f17a-5f7f9db77d-hlc8c" event={"ID":"7f7cbb3b-d4d4-4523-a596-52c9fe90aad0","Type":"ContainerStarted","Data":"d582214d0884f7b06165707e1c962670f7aa9d627091804ccd2a84e21c51501e"}
Oct 30 22:50:11.998254 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:50:11.997845    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8756f17a-5f7f9db77d-hlc8c"
Oct 30 22:50:11.999415 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:50:11.999391    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8756f17a-7cddb85964-lnvkn" event={"ID":"1da7d0be-1c41-4da5-b4f4-093295e92841","Type":"ContainerStarted","Data":"162bcc3010d8853f0b238065d1139f12c81b2c724db7772bdafb1a004644b7f1"}
Oct 30 22:50:11.999578 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:50:11.999432    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-8756f17a-7cddb85964-lnvkn"
Oct 30 22:50:11.999578 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:50:11.999448    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8756f17a-7cddb85964-lnvkn" event={"ID":"1da7d0be-1c41-4da5-b4f4-093295e92841","Type":"ContainerStarted","Data":"b988de2324da871f8c5739687be4920509697aa1c6b5a57030abccaf9ddb13de"}
Oct 30 22:50:12.121313 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:50:12.121258    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8756f17a-5f7f9db77d-hlc8c" podStartSLOduration=2.219591585 podStartE2EDuration="3.121244452s" totalImagesPullingTime="901.652867ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:50:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:50:12.119965056 +0000 UTC m=+4065.732734813" watchObservedRunningTime="2026-10-30 22:50:12.121244452 +0000 UTC m=+4065.734014209"
Oct 30 22:50:12.249927 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:50:12.249861    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-8756f17a-7cddb85964-lnvkn" podStartSLOduration=2.24984389 podStartE2EDuration="2.24984389s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:50:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:50:12.249180193 +0000 UTC m=+4065.861949979" watchObservedRunningTime="2026-10-30 22:50:12.24984389 +0000 UTC m=+4065.862613649"
Oct 30 22:50:18.009848 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:50:18.009816    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8756f17a-5f7f9db77d-hlc8c"
Oct 30 22:50:23.007166 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:50:23.007131    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-8756f17a-7cddb85964-lnvkn"
Oct 30 22:50:24.666304 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:50:24.666269    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 22:50:49.365010 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:50:49.364933    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8756f17a-7cddb85964-lnvkn"]
Oct 30 22:50:54.366249 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:50:54.366208    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-8756f17a-7cddb85964-lnvkn" podUID="1da7d0be-1c41-4da5-b4f4-093295e92841" containerName="payload-processing" containerID="cri-o://162bcc3010d8853f0b238065d1139f12c81b2c724db7772bdafb1a004644b7f1" gracePeriod=25
Oct 30 22:50:54.622402 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:50:54.622351    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8756f17a-7cddb85964-lnvkn"
Oct 30 22:50:54.666127 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:50:54.666080    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 22:50:54.672742 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:50:54.672719    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1da7d0be-1c41-4da5-b4f4-093295e92841-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1da7d0be-1c41-4da5-b4f4-093295e92841-plugins-config-volume\") pod \"1da7d0be-1c41-4da5-b4f4-093295e92841\" (UID: \"1da7d0be-1c41-4da5-b4f4-093295e92841\") "
Oct 30 22:50:54.672841 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:50:54.672778    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1da7d0be-1c41-4da5-b4f4-093295e92841-kube-api-access-cbmgx\" (UniqueName: \"kubernetes.io/projected/1da7d0be-1c41-4da5-b4f4-093295e92841-kube-api-access-cbmgx\") pod \"1da7d0be-1c41-4da5-b4f4-093295e92841\" (UID: \"1da7d0be-1c41-4da5-b4f4-093295e92841\") "
Oct 30 22:50:54.673071 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:50:54.673047    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1da7d0be-1c41-4da5-b4f4-093295e92841-plugins-config-volume" pod "1da7d0be-1c41-4da5-b4f4-093295e92841" (UID: "1da7d0be-1c41-4da5-b4f4-093295e92841"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:50:54.673189 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:50:54.673175    5061 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1da7d0be-1c41-4da5-b4f4-093295e92841-plugins-config-volume\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:50:54.674849 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:50:54.674826    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1da7d0be-1c41-4da5-b4f4-093295e92841-kube-api-access-cbmgx" pod "1da7d0be-1c41-4da5-b4f4-093295e92841" (UID: "1da7d0be-1c41-4da5-b4f4-093295e92841"). InnerVolumeSpecName "kube-api-access-cbmgx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:50:54.773942 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:50:54.773909    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbmgx\" (UniqueName: \"kubernetes.io/projected/1da7d0be-1c41-4da5-b4f4-093295e92841-kube-api-access-cbmgx\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:50:55.234763 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:50:55.234730    5061 generic.go:356] "Generic (PLEG): container finished" podID="1da7d0be-1c41-4da5-b4f4-093295e92841" containerID="162bcc3010d8853f0b238065d1139f12c81b2c724db7772bdafb1a004644b7f1" exitCode=0
Oct 30 22:50:55.234926 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:50:55.234771    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8756f17a-7cddb85964-lnvkn"
Oct 30 22:50:55.234926 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:50:55.234804    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8756f17a-7cddb85964-lnvkn" event={"ID":"1da7d0be-1c41-4da5-b4f4-093295e92841","Type":"ContainerDied","Data":"162bcc3010d8853f0b238065d1139f12c81b2c724db7772bdafb1a004644b7f1"}
Oct 30 22:50:55.234926 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:50:55.234852    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8756f17a-7cddb85964-lnvkn" event={"ID":"1da7d0be-1c41-4da5-b4f4-093295e92841","Type":"ContainerDied","Data":"b988de2324da871f8c5739687be4920509697aa1c6b5a57030abccaf9ddb13de"}
Oct 30 22:50:55.234926 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:50:55.234875    5061 scope.go:122] "RemoveContainer" containerID="162bcc3010d8853f0b238065d1139f12c81b2c724db7772bdafb1a004644b7f1"
Oct 30 22:50:55.244662 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:50:55.244645    5061 scope.go:122] "RemoveContainer" containerID="162bcc3010d8853f0b238065d1139f12c81b2c724db7772bdafb1a004644b7f1"
Oct 30 22:50:55.244923 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:50:55.244903    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"162bcc3010d8853f0b238065d1139f12c81b2c724db7772bdafb1a004644b7f1\": container with ID starting with 162bcc3010d8853f0b238065d1139f12c81b2c724db7772bdafb1a004644b7f1 not found: ID does not exist" containerID="162bcc3010d8853f0b238065d1139f12c81b2c724db7772bdafb1a004644b7f1"
Oct 30 22:50:55.245003 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:50:55.244930    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"162bcc3010d8853f0b238065d1139f12c81b2c724db7772bdafb1a004644b7f1"} err="failed to get container status \"162bcc3010d8853f0b238065d1139f12c81b2c724db7772bdafb1a004644b7f1\": rpc error: code = NotFound desc = could not find container \"162bcc3010d8853f0b238065d1139f12c81b2c724db7772bdafb1a004644b7f1\": container with ID starting with 162bcc3010d8853f0b238065d1139f12c81b2c724db7772bdafb1a004644b7f1 not found: ID does not exist"
Oct 30 22:50:55.407600 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:50:55.407571    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8756f17a-7cddb85964-lnvkn"]
Oct 30 22:50:55.459759 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:50:55.459734    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8756f17a-7cddb85964-lnvkn"]
Oct 30 22:50:55.587267 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:50:55.587238    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8756f17a-5f7f9db77d-hlc8c"]
Oct 30 22:50:55.587455 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:50:55.587424    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8756f17a-5f7f9db77d-hlc8c" podUID="7f7cbb3b-d4d4-4523-a596-52c9fe90aad0" containerName="maas-api" containerID="cri-o://d582214d0884f7b06165707e1c962670f7aa9d627091804ccd2a84e21c51501e" gracePeriod=30
Oct 30 22:50:55.844890 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:50:55.844827    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8756f17a-5f7f9db77d-hlc8c"
Oct 30 22:50:55.984948 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:50:55.984915    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f7cbb3b-d4d4-4523-a596-52c9fe90aad0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f7cbb3b-d4d4-4523-a596-52c9fe90aad0-maas-api-tls\") pod \"7f7cbb3b-d4d4-4523-a596-52c9fe90aad0\" (UID: \"7f7cbb3b-d4d4-4523-a596-52c9fe90aad0\") "
Oct 30 22:50:55.985120 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:50:55.985018    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f7cbb3b-d4d4-4523-a596-52c9fe90aad0-kube-api-access-xrtjg\" (UniqueName: \"kubernetes.io/projected/7f7cbb3b-d4d4-4523-a596-52c9fe90aad0-kube-api-access-xrtjg\") pod \"7f7cbb3b-d4d4-4523-a596-52c9fe90aad0\" (UID: \"7f7cbb3b-d4d4-4523-a596-52c9fe90aad0\") "
Oct 30 22:50:55.985186 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:50:55.985171    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f7cbb3b-d4d4-4523-a596-52c9fe90aad0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f7cbb3b-d4d4-4523-a596-52c9fe90aad0-metrics-certs\") pod \"7f7cbb3b-d4d4-4523-a596-52c9fe90aad0\" (UID: \"7f7cbb3b-d4d4-4523-a596-52c9fe90aad0\") "
Oct 30 22:50:55.985229 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:50:55.985214    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f7cbb3b-d4d4-4523-a596-52c9fe90aad0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f7cbb3b-d4d4-4523-a596-52c9fe90aad0-openshift-service-ca\") pod \"7f7cbb3b-d4d4-4523-a596-52c9fe90aad0\" (UID: \"7f7cbb3b-d4d4-4523-a596-52c9fe90aad0\") "
Oct 30 22:50:55.985669 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:50:55.985638    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f7cbb3b-d4d4-4523-a596-52c9fe90aad0-openshift-service-ca" pod "7f7cbb3b-d4d4-4523-a596-52c9fe90aad0" (UID: "7f7cbb3b-d4d4-4523-a596-52c9fe90aad0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:50:55.987266 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:50:55.987239    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f7cbb3b-d4d4-4523-a596-52c9fe90aad0-metrics-certs" pod "7f7cbb3b-d4d4-4523-a596-52c9fe90aad0" (UID: "7f7cbb3b-d4d4-4523-a596-52c9fe90aad0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:50:55.987576 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:50:55.987552    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f7cbb3b-d4d4-4523-a596-52c9fe90aad0-maas-api-tls" pod "7f7cbb3b-d4d4-4523-a596-52c9fe90aad0" (UID: "7f7cbb3b-d4d4-4523-a596-52c9fe90aad0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:50:55.987642 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:50:55.987556    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f7cbb3b-d4d4-4523-a596-52c9fe90aad0-kube-api-access-xrtjg" pod "7f7cbb3b-d4d4-4523-a596-52c9fe90aad0" (UID: "7f7cbb3b-d4d4-4523-a596-52c9fe90aad0"). InnerVolumeSpecName "kube-api-access-xrtjg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:50:56.087011 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:50:56.086976    5061 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f7cbb3b-d4d4-4523-a596-52c9fe90aad0-metrics-certs\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:50:56.087011 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:50:56.087009    5061 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f7cbb3b-d4d4-4523-a596-52c9fe90aad0-openshift-service-ca\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:50:56.087278 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:50:56.087023    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f7cbb3b-d4d4-4523-a596-52c9fe90aad0-maas-api-tls\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:50:56.087278 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:50:56.087036    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrtjg\" (UniqueName: \"kubernetes.io/projected/7f7cbb3b-d4d4-4523-a596-52c9fe90aad0-kube-api-access-xrtjg\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:50:56.240882 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:50:56.240794    5061 generic.go:356] "Generic (PLEG): container finished" podID="7f7cbb3b-d4d4-4523-a596-52c9fe90aad0" containerID="d582214d0884f7b06165707e1c962670f7aa9d627091804ccd2a84e21c51501e" exitCode=0
Oct 30 22:50:56.240882 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:50:56.240843    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8756f17a-5f7f9db77d-hlc8c"
Oct 30 22:50:56.241120 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:50:56.240878    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8756f17a-5f7f9db77d-hlc8c" event={"ID":"7f7cbb3b-d4d4-4523-a596-52c9fe90aad0","Type":"ContainerDied","Data":"d582214d0884f7b06165707e1c962670f7aa9d627091804ccd2a84e21c51501e"}
Oct 30 22:50:56.241120 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:50:56.240942    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8756f17a-5f7f9db77d-hlc8c" event={"ID":"7f7cbb3b-d4d4-4523-a596-52c9fe90aad0","Type":"ContainerDied","Data":"1a82be054653ed2e59e2e8d5bb1c47ebfe6b9a42ea1b8612cd7e82d47f36d499"}
Oct 30 22:50:56.241120 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:50:56.240970    5061 scope.go:122] "RemoveContainer" containerID="d582214d0884f7b06165707e1c962670f7aa9d627091804ccd2a84e21c51501e"
Oct 30 22:50:56.251037 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:50:56.251020    5061 scope.go:122] "RemoveContainer" containerID="d582214d0884f7b06165707e1c962670f7aa9d627091804ccd2a84e21c51501e"
Oct 30 22:50:56.251342 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:50:56.251324    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d582214d0884f7b06165707e1c962670f7aa9d627091804ccd2a84e21c51501e\": container with ID starting with d582214d0884f7b06165707e1c962670f7aa9d627091804ccd2a84e21c51501e not found: ID does not exist" containerID="d582214d0884f7b06165707e1c962670f7aa9d627091804ccd2a84e21c51501e"
Oct 30 22:50:56.251410 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:50:56.251348    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d582214d0884f7b06165707e1c962670f7aa9d627091804ccd2a84e21c51501e"} err="failed to get container status \"d582214d0884f7b06165707e1c962670f7aa9d627091804ccd2a84e21c51501e\": rpc error: code = NotFound desc = could not find container \"d582214d0884f7b06165707e1c962670f7aa9d627091804ccd2a84e21c51501e\": container with ID starting with d582214d0884f7b06165707e1c962670f7aa9d627091804ccd2a84e21c51501e not found: ID does not exist"
Oct 30 22:50:56.413118 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:50:56.413086    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8756f17a-5f7f9db77d-hlc8c"]
Oct 30 22:50:56.467663 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:50:56.467621    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8756f17a-5f7f9db77d-hlc8c"]
Oct 30 22:50:56.526999 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:50:56.526922    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1da7d0be-1c41-4da5-b4f4-093295e92841" path="/var/lib/kubelet/pods/1da7d0be-1c41-4da5-b4f4-093295e92841/volumes"
Oct 30 22:50:56.527357 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:50:56.527344    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f7cbb3b-d4d4-4523-a596-52c9fe90aad0" path="/var/lib/kubelet/pods/7f7cbb3b-d4d4-4523-a596-52c9fe90aad0/volumes"
Oct 30 22:51:01.041255 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:51:01.041223    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-authcfg-http-8a1d54-openshift-default-6f54ddfbb5-rx7tn"]
Oct 30 22:51:01.041695 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:51:01.041654    5061 cpu_manager.go:414] "containerMap: removing container" podUID="1da7d0be-1c41-4da5-b4f4-093295e92841" containerName="payload-processing"
Oct 30 22:51:01.041695 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:51:01.041666    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1da7d0be-1c41-4da5-b4f4-093295e92841" containerName="payload-processing"
Oct 30 22:51:01.041784 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:51:01.041698    5061 cpu_manager.go:414] "containerMap: removing container" podUID="7f7cbb3b-d4d4-4523-a596-52c9fe90aad0" containerName="maas-api"
Oct 30 22:51:01.041784 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:51:01.041704    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f7cbb3b-d4d4-4523-a596-52c9fe90aad0" containerName="maas-api"
Oct 30 22:51:01.041784 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:51:01.041769    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f7cbb3b-d4d4-4523-a596-52c9fe90aad0" containerName="maas-api"
Oct 30 22:51:01.041784 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:51:01.041781    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="1da7d0be-1c41-4da5-b4f4-093295e92841" containerName="payload-processing"
Oct 30 22:51:01.046292 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:51:01.046274    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-8a1d54-openshift-default-6f54ddfbb5-rx7tn"
Oct 30 22:51:01.101997 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:51:01.101968    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-authcfg-http-8a1d54-openshift-default-dockercfg-6tbhz\""
Oct 30 22:51:01.139985 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:51:01.139954    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe39287f-2cc6-4335-8df5-f74fa55515a1-workload-certs\") pod \"e2e-authcfg-http-8a1d54-openshift-default-6f54ddfbb5-rx7tn\" (UID: \"fe39287f-2cc6-4335-8df5-f74fa55515a1\") " pod="openshift-ingress/e2e-authcfg-http-8a1d54-openshift-default-6f54ddfbb5-rx7tn"
Oct 30 22:51:01.140139 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:51:01.139995    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe39287f-2cc6-4335-8df5-f74fa55515a1-istio-envoy\") pod \"e2e-authcfg-http-8a1d54-openshift-default-6f54ddfbb5-rx7tn\" (UID: \"fe39287f-2cc6-4335-8df5-f74fa55515a1\") " pod="openshift-ingress/e2e-authcfg-http-8a1d54-openshift-default-6f54ddfbb5-rx7tn"
Oct 30 22:51:01.140139 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:51:01.140033    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe39287f-2cc6-4335-8df5-f74fa55515a1-istio-podinfo\") pod \"e2e-authcfg-http-8a1d54-openshift-default-6f54ddfbb5-rx7tn\" (UID: \"fe39287f-2cc6-4335-8df5-f74fa55515a1\") " pod="openshift-ingress/e2e-authcfg-http-8a1d54-openshift-default-6f54ddfbb5-rx7tn"
Oct 30 22:51:01.140217 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:51:01.140170    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe39287f-2cc6-4335-8df5-f74fa55515a1-istio-data\") pod \"e2e-authcfg-http-8a1d54-openshift-default-6f54ddfbb5-rx7tn\" (UID: \"fe39287f-2cc6-4335-8df5-f74fa55515a1\") " pod="openshift-ingress/e2e-authcfg-http-8a1d54-openshift-default-6f54ddfbb5-rx7tn"
Oct 30 22:51:01.140250 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:51:01.140217    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe39287f-2cc6-4335-8df5-f74fa55515a1-credential-socket\") pod \"e2e-authcfg-http-8a1d54-openshift-default-6f54ddfbb5-rx7tn\" (UID: \"fe39287f-2cc6-4335-8df5-f74fa55515a1\") " pod="openshift-ingress/e2e-authcfg-http-8a1d54-openshift-default-6f54ddfbb5-rx7tn"
Oct 30 22:51:01.140339 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:51:01.140324    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkvx2\" (UniqueName: \"kubernetes.io/projected/fe39287f-2cc6-4335-8df5-f74fa55515a1-kube-api-access-hkvx2\") pod \"e2e-authcfg-http-8a1d54-openshift-default-6f54ddfbb5-rx7tn\" (UID: \"fe39287f-2cc6-4335-8df5-f74fa55515a1\") " pod="openshift-ingress/e2e-authcfg-http-8a1d54-openshift-default-6f54ddfbb5-rx7tn"
Oct 30 22:51:01.140376 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:51:01.140349    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe39287f-2cc6-4335-8df5-f74fa55515a1-istio-token\") pod \"e2e-authcfg-http-8a1d54-openshift-default-6f54ddfbb5-rx7tn\" (UID: \"fe39287f-2cc6-4335-8df5-f74fa55515a1\") " pod="openshift-ingress/e2e-authcfg-http-8a1d54-openshift-default-6f54ddfbb5-rx7tn"
Oct 30 22:51:01.140411 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:51:01.140375    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe39287f-2cc6-4335-8df5-f74fa55515a1-workload-socket\") pod \"e2e-authcfg-http-8a1d54-openshift-default-6f54ddfbb5-rx7tn\" (UID: \"fe39287f-2cc6-4335-8df5-f74fa55515a1\") " pod="openshift-ingress/e2e-authcfg-http-8a1d54-openshift-default-6f54ddfbb5-rx7tn"
Oct 30 22:51:01.140444 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:51:01.140422    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe39287f-2cc6-4335-8df5-f74fa55515a1-istiod-ca-cert\") pod \"e2e-authcfg-http-8a1d54-openshift-default-6f54ddfbb5-rx7tn\" (UID: \"fe39287f-2cc6-4335-8df5-f74fa55515a1\") " pod="openshift-ingress/e2e-authcfg-http-8a1d54-openshift-default-6f54ddfbb5-rx7tn"
Oct 30 22:51:01.156774 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:51:01.156745    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-8a1d54-openshift-default-6f54ddfbb5-rx7tn"]
Oct 30 22:51:01.241125 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:51:01.241085    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe39287f-2cc6-4335-8df5-f74fa55515a1-workload-certs\") pod \"e2e-authcfg-http-8a1d54-openshift-default-6f54ddfbb5-rx7tn\" (UID: \"fe39287f-2cc6-4335-8df5-f74fa55515a1\") " pod="openshift-ingress/e2e-authcfg-http-8a1d54-openshift-default-6f54ddfbb5-rx7tn"
Oct 30 22:51:01.241125 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:51:01.241129    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe39287f-2cc6-4335-8df5-f74fa55515a1-istio-envoy\") pod \"e2e-authcfg-http-8a1d54-openshift-default-6f54ddfbb5-rx7tn\" (UID: \"fe39287f-2cc6-4335-8df5-f74fa55515a1\") " pod="openshift-ingress/e2e-authcfg-http-8a1d54-openshift-default-6f54ddfbb5-rx7tn"
Oct 30 22:51:01.241387 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:51:01.241144    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe39287f-2cc6-4335-8df5-f74fa55515a1-istio-podinfo\") pod \"e2e-authcfg-http-8a1d54-openshift-default-6f54ddfbb5-rx7tn\" (UID: \"fe39287f-2cc6-4335-8df5-f74fa55515a1\") " pod="openshift-ingress/e2e-authcfg-http-8a1d54-openshift-default-6f54ddfbb5-rx7tn"
Oct 30 22:51:01.241387 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:51:01.241167    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe39287f-2cc6-4335-8df5-f74fa55515a1-istio-data\") pod \"e2e-authcfg-http-8a1d54-openshift-default-6f54ddfbb5-rx7tn\" (UID: \"fe39287f-2cc6-4335-8df5-f74fa55515a1\") " pod="openshift-ingress/e2e-authcfg-http-8a1d54-openshift-default-6f54ddfbb5-rx7tn"
Oct 30 22:51:01.241387 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:51:01.241199    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe39287f-2cc6-4335-8df5-f74fa55515a1-credential-socket\") pod \"e2e-authcfg-http-8a1d54-openshift-default-6f54ddfbb5-rx7tn\" (UID: \"fe39287f-2cc6-4335-8df5-f74fa55515a1\") " pod="openshift-ingress/e2e-authcfg-http-8a1d54-openshift-default-6f54ddfbb5-rx7tn"
Oct 30 22:51:01.241387 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:51:01.241250    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkvx2\" (UniqueName: \"kubernetes.io/projected/fe39287f-2cc6-4335-8df5-f74fa55515a1-kube-api-access-hkvx2\") pod \"e2e-authcfg-http-8a1d54-openshift-default-6f54ddfbb5-rx7tn\" (UID: \"fe39287f-2cc6-4335-8df5-f74fa55515a1\") " pod="openshift-ingress/e2e-authcfg-http-8a1d54-openshift-default-6f54ddfbb5-rx7tn"
Oct 30 22:51:01.241656 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:51:01.241392    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe39287f-2cc6-4335-8df5-f74fa55515a1-istio-token\") pod \"e2e-authcfg-http-8a1d54-openshift-default-6f54ddfbb5-rx7tn\" (UID: \"fe39287f-2cc6-4335-8df5-f74fa55515a1\") " pod="openshift-ingress/e2e-authcfg-http-8a1d54-openshift-default-6f54ddfbb5-rx7tn"
Oct 30 22:51:01.241656 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:51:01.241430    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe39287f-2cc6-4335-8df5-f74fa55515a1-workload-socket\") pod \"e2e-authcfg-http-8a1d54-openshift-default-6f54ddfbb5-rx7tn\" (UID: \"fe39287f-2cc6-4335-8df5-f74fa55515a1\") " pod="openshift-ingress/e2e-authcfg-http-8a1d54-openshift-default-6f54ddfbb5-rx7tn"
Oct 30 22:51:01.241656 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:51:01.241459    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe39287f-2cc6-4335-8df5-f74fa55515a1-istiod-ca-cert\") pod \"e2e-authcfg-http-8a1d54-openshift-default-6f54ddfbb5-rx7tn\" (UID: \"fe39287f-2cc6-4335-8df5-f74fa55515a1\") " pod="openshift-ingress/e2e-authcfg-http-8a1d54-openshift-default-6f54ddfbb5-rx7tn"
Oct 30 22:51:01.241835 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:51:01.241687    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe39287f-2cc6-4335-8df5-f74fa55515a1-workload-certs\") pod \"e2e-authcfg-http-8a1d54-openshift-default-6f54ddfbb5-rx7tn\" (UID: \"fe39287f-2cc6-4335-8df5-f74fa55515a1\") " pod="openshift-ingress/e2e-authcfg-http-8a1d54-openshift-default-6f54ddfbb5-rx7tn"
Oct 30 22:51:01.241835 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:51:01.241766    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe39287f-2cc6-4335-8df5-f74fa55515a1-credential-socket\") pod \"e2e-authcfg-http-8a1d54-openshift-default-6f54ddfbb5-rx7tn\" (UID: \"fe39287f-2cc6-4335-8df5-f74fa55515a1\") " pod="openshift-ingress/e2e-authcfg-http-8a1d54-openshift-default-6f54ddfbb5-rx7tn"
Oct 30 22:51:01.241956 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:51:01.241880    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe39287f-2cc6-4335-8df5-f74fa55515a1-istio-data\") pod \"e2e-authcfg-http-8a1d54-openshift-default-6f54ddfbb5-rx7tn\" (UID: \"fe39287f-2cc6-4335-8df5-f74fa55515a1\") " pod="openshift-ingress/e2e-authcfg-http-8a1d54-openshift-default-6f54ddfbb5-rx7tn"
Oct 30 22:51:01.241956 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:51:01.241925    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe39287f-2cc6-4335-8df5-f74fa55515a1-workload-socket\") pod \"e2e-authcfg-http-8a1d54-openshift-default-6f54ddfbb5-rx7tn\" (UID: \"fe39287f-2cc6-4335-8df5-f74fa55515a1\") " pod="openshift-ingress/e2e-authcfg-http-8a1d54-openshift-default-6f54ddfbb5-rx7tn"
Oct 30 22:51:01.242323 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:51:01.242301    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe39287f-2cc6-4335-8df5-f74fa55515a1-istiod-ca-cert\") pod \"e2e-authcfg-http-8a1d54-openshift-default-6f54ddfbb5-rx7tn\" (UID: \"fe39287f-2cc6-4335-8df5-f74fa55515a1\") " pod="openshift-ingress/e2e-authcfg-http-8a1d54-openshift-default-6f54ddfbb5-rx7tn"
Oct 30 22:51:01.243614 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:51:01.243593    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe39287f-2cc6-4335-8df5-f74fa55515a1-istio-envoy\") pod \"e2e-authcfg-http-8a1d54-openshift-default-6f54ddfbb5-rx7tn\" (UID: \"fe39287f-2cc6-4335-8df5-f74fa55515a1\") " pod="openshift-ingress/e2e-authcfg-http-8a1d54-openshift-default-6f54ddfbb5-rx7tn"
Oct 30 22:51:01.243805 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:51:01.243789    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe39287f-2cc6-4335-8df5-f74fa55515a1-istio-podinfo\") pod \"e2e-authcfg-http-8a1d54-openshift-default-6f54ddfbb5-rx7tn\" (UID: \"fe39287f-2cc6-4335-8df5-f74fa55515a1\") " pod="openshift-ingress/e2e-authcfg-http-8a1d54-openshift-default-6f54ddfbb5-rx7tn"
Oct 30 22:51:01.303934 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:51:01.303902    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe39287f-2cc6-4335-8df5-f74fa55515a1-istio-token\") pod \"e2e-authcfg-http-8a1d54-openshift-default-6f54ddfbb5-rx7tn\" (UID: \"fe39287f-2cc6-4335-8df5-f74fa55515a1\") " pod="openshift-ingress/e2e-authcfg-http-8a1d54-openshift-default-6f54ddfbb5-rx7tn"
Oct 30 22:51:01.304192 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:51:01.304174    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkvx2\" (UniqueName: \"kubernetes.io/projected/fe39287f-2cc6-4335-8df5-f74fa55515a1-kube-api-access-hkvx2\") pod \"e2e-authcfg-http-8a1d54-openshift-default-6f54ddfbb5-rx7tn\" (UID: \"fe39287f-2cc6-4335-8df5-f74fa55515a1\") " pod="openshift-ingress/e2e-authcfg-http-8a1d54-openshift-default-6f54ddfbb5-rx7tn"
Oct 30 22:51:01.359499 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:51:01.359458    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-8a1d54-openshift-default-6f54ddfbb5-rx7tn"
Oct 30 22:51:01.778539 ip-10-0-130-60 kubenswrapper[5061]: W1030 22:51:01.778494    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe39287f_2cc6_4335_8df5_f74fa55515a1.slice/crio-35ede583c9dd0340626215d173233df35f5d3d4b56d706d592b39f48bd126c9b WatchSource:0}: Error finding container 35ede583c9dd0340626215d173233df35f5d3d4b56d706d592b39f48bd126c9b: Status 404 returned error can't find the container with id 35ede583c9dd0340626215d173233df35f5d3d4b56d706d592b39f48bd126c9b
Oct 30 22:51:01.781784 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:51:01.781760    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-8a1d54-openshift-default-6f54ddfbb5-rx7tn"]
Oct 30 22:51:01.782865 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:51:01.782835    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:51:01.782939 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:51:01.782895    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:51:01.782939 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:51:01.782922    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:51:02.278166 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:51:02.278132    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-8a1d54-openshift-default-6f54ddfbb5-rx7tn" event={"ID":"fe39287f-2cc6-4335-8df5-f74fa55515a1","Type":"ContainerStarted","Data":"5bfcc39a2867af7ef9b9abd6f4d69835a5faabb49c869c7c1651d3a169555dad"}
Oct 30 22:51:02.278621 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:51:02.278187    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-8a1d54-openshift-default-6f54ddfbb5-rx7tn" event={"ID":"fe39287f-2cc6-4335-8df5-f74fa55515a1","Type":"ContainerStarted","Data":"35ede583c9dd0340626215d173233df35f5d3d4b56d706d592b39f48bd126c9b"}
Oct 30 22:51:02.359803 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:51:02.359778    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-8a1d54-openshift-default-6f54ddfbb5-rx7tn"
Oct 30 22:51:02.364753 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:51:02.364728    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-8a1d54-openshift-default-6f54ddfbb5-rx7tn"
Oct 30 22:51:02.402931 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:51:02.402886    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-authcfg-http-8a1d54-openshift-default-6f54ddfbb5-rx7tn" podStartSLOduration=2.402874688 podStartE2EDuration="2.402874688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:51:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:51:02.402142999 +0000 UTC m=+4116.014912758" watchObservedRunningTime="2026-10-30 22:51:02.402874688 +0000 UTC m=+4116.015644447"
Oct 30 22:51:03.284101 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:51:03.284067    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-8a1d54-openshift-default-6f54ddfbb5-rx7tn"
Oct 30 22:51:03.284973 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:51:03.284950    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-8a1d54-openshift-default-6f54ddfbb5-rx7tn"
Oct 30 22:51:24.666127 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:51:24.666084    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 22:51:28.429665 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:51:28.429602    5061 generic.go:356] "Generic (PLEG): container finished" podID="fe39287f-2cc6-4335-8df5-f74fa55515a1" containerID="5bfcc39a2867af7ef9b9abd6f4d69835a5faabb49c869c7c1651d3a169555dad" exitCode=0
Oct 30 22:51:28.430030 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:51:28.429678    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-8a1d54-openshift-default-6f54ddfbb5-rx7tn" event={"ID":"fe39287f-2cc6-4335-8df5-f74fa55515a1","Type":"ContainerDied","Data":"5bfcc39a2867af7ef9b9abd6f4d69835a5faabb49c869c7c1651d3a169555dad"}
Oct 30 22:51:28.430077 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:51:28.430025    5061 scope.go:122] "RemoveContainer" containerID="5bfcc39a2867af7ef9b9abd6f4d69835a5faabb49c869c7c1651d3a169555dad"
Oct 30 22:51:28.431211 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:51:28.431180    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:51:28.431270 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:51:28.431254    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:51:28.431306 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:51:28.431284    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:51:29.439756 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:51:29.439715    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-8a1d54-openshift-default-6f54ddfbb5-rx7tn" event={"ID":"fe39287f-2cc6-4335-8df5-f74fa55515a1","Type":"ContainerStarted","Data":"05225f741fe8025afd6eb6fc27c0255fb0164b5231281e4624fe3dbc172bb17b"}
Oct 30 22:51:30.360169 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:51:30.360080    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-8a1d54-openshift-default-6f54ddfbb5-rx7tn"
Oct 30 22:51:30.361047 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:51:30.361026    5061 patch_prober.go:29] interesting pod/e2e-authcfg-http-8a1d54-openshift-default-6f54ddfbb5-rx7tn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" start-of-body=
Oct 30 22:51:30.361130 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:51:30.361084    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-8a1d54-openshift-default-6f54ddfbb5-rx7tn" podUID="fe39287f-2cc6-4335-8df5-f74fa55515a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused"
Oct 30 22:51:31.360593 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:51:31.360555    5061 patch_prober.go:29] interesting pod/e2e-authcfg-http-8a1d54-openshift-default-6f54ddfbb5-rx7tn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" start-of-body=
Oct 30 22:51:31.361004 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:51:31.360618    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-8a1d54-openshift-default-6f54ddfbb5-rx7tn" podUID="fe39287f-2cc6-4335-8df5-f74fa55515a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused"
Oct 30 22:51:32.359933 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:51:32.359892    5061 patch_prober.go:29] interesting pod/e2e-authcfg-http-8a1d54-openshift-default-6f54ddfbb5-rx7tn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" start-of-body=
Oct 30 22:51:32.360132 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:51:32.359951    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-8a1d54-openshift-default-6f54ddfbb5-rx7tn" podUID="fe39287f-2cc6-4335-8df5-f74fa55515a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused"
Oct 30 22:51:33.285079 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:51:33.285046    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-8a1d54-openshift-default-6f54ddfbb5-rx7tn"
Oct 30 22:51:33.359869 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:51:33.359834    5061 patch_prober.go:29] interesting pod/e2e-authcfg-http-8a1d54-openshift-default-6f54ddfbb5-rx7tn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" start-of-body=
Oct 30 22:51:33.360031 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:51:33.359896    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-8a1d54-openshift-default-6f54ddfbb5-rx7tn" podUID="fe39287f-2cc6-4335-8df5-f74fa55515a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused"
Oct 30 22:51:34.360077 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:51:34.360043    5061 patch_prober.go:29] interesting pod/e2e-authcfg-http-8a1d54-openshift-default-6f54ddfbb5-rx7tn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" start-of-body=
Oct 30 22:51:34.360472 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:51:34.360097    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-8a1d54-openshift-default-6f54ddfbb5-rx7tn" podUID="fe39287f-2cc6-4335-8df5-f74fa55515a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused"
Oct 30 22:51:35.364284 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:51:35.364253    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-8a1d54-openshift-default-6f54ddfbb5-rx7tn"
Oct 30 22:51:35.365432 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:51:35.365413    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-8a1d54-openshift-default-6f54ddfbb5-rx7tn"
Oct 30 22:51:54.665901 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:51:54.665863    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 22:51:57.052907 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:51:57.052874    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1efcb985-hchct"]
Oct 30 22:51:57.105153 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:51:57.105123    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1efcb985-hchct"]
Oct 30 22:51:58.527208 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:51:58.527167    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db90a23a-4a63-490b-8e3c-4fe6e50f1df7" path="/var/lib/kubelet/pods/db90a23a-4a63-490b-8e3c-4fe6e50f1df7/volumes"
Oct 30 22:52:24.673806 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:52:24.673725    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 22:52:27.101216 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:52:27.101183    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-sclrt_5e74071e-0462-42b5-a05c-d90bb638478a/console-operator/2.log"
Oct 30 22:52:27.145155 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:52:27.145131    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-sclrt_5e74071e-0462-42b5-a05c-d90bb638478a/console-operator/2.log"
Oct 30 22:52:54.667350 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:52:54.667305    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 22:53:22.784365 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:53:22.784328    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-8a1d54-openshift-default-6f54ddfbb5-rx7tn"]
Oct 30 22:53:22.784914 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:53:22.784560    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-authcfg-http-8a1d54-openshift-default-6f54ddfbb5-rx7tn" podUID="fe39287f-2cc6-4335-8df5-f74fa55515a1" containerName="istio-proxy" containerID="cri-o://05225f741fe8025afd6eb6fc27c0255fb0164b5231281e4624fe3dbc172bb17b" gracePeriod=30
Oct 30 22:53:24.669679 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:53:24.669638    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 22:53:28.048680 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:53:28.048652    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-8a1d54-openshift-default-6f54ddfbb5-rx7tn"
Oct 30 22:53:28.054580 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:53:28.054544    5061 generic.go:356] "Generic (PLEG): container finished" podID="fe39287f-2cc6-4335-8df5-f74fa55515a1" containerID="05225f741fe8025afd6eb6fc27c0255fb0164b5231281e4624fe3dbc172bb17b" exitCode=0
Oct 30 22:53:28.054696 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:53:28.054581    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-8a1d54-openshift-default-6f54ddfbb5-rx7tn" event={"ID":"fe39287f-2cc6-4335-8df5-f74fa55515a1","Type":"ContainerDied","Data":"05225f741fe8025afd6eb6fc27c0255fb0164b5231281e4624fe3dbc172bb17b"}
Oct 30 22:53:28.054696 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:53:28.054607    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-8a1d54-openshift-default-6f54ddfbb5-rx7tn"
Oct 30 22:53:28.054696 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:53:28.054647    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-8a1d54-openshift-default-6f54ddfbb5-rx7tn" event={"ID":"fe39287f-2cc6-4335-8df5-f74fa55515a1","Type":"ContainerDied","Data":"35ede583c9dd0340626215d173233df35f5d3d4b56d706d592b39f48bd126c9b"}
Oct 30 22:53:28.054696 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:53:28.054681    5061 scope.go:122] "RemoveContainer" containerID="05225f741fe8025afd6eb6fc27c0255fb0164b5231281e4624fe3dbc172bb17b"
Oct 30 22:53:28.063391 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:53:28.063375    5061 scope.go:122] "RemoveContainer" containerID="5bfcc39a2867af7ef9b9abd6f4d69835a5faabb49c869c7c1651d3a169555dad"
Oct 30 22:53:28.072025 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:53:28.072008    5061 scope.go:122] "RemoveContainer" containerID="05225f741fe8025afd6eb6fc27c0255fb0164b5231281e4624fe3dbc172bb17b"
Oct 30 22:53:28.072284 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:53:28.072267    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05225f741fe8025afd6eb6fc27c0255fb0164b5231281e4624fe3dbc172bb17b\": container with ID starting with 05225f741fe8025afd6eb6fc27c0255fb0164b5231281e4624fe3dbc172bb17b not found: ID does not exist" containerID="05225f741fe8025afd6eb6fc27c0255fb0164b5231281e4624fe3dbc172bb17b"
Oct 30 22:53:28.072338 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:53:28.072297    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05225f741fe8025afd6eb6fc27c0255fb0164b5231281e4624fe3dbc172bb17b"} err="failed to get container status \"05225f741fe8025afd6eb6fc27c0255fb0164b5231281e4624fe3dbc172bb17b\": rpc error: code = NotFound desc = could not find container \"05225f741fe8025afd6eb6fc27c0255fb0164b5231281e4624fe3dbc172bb17b\": container with ID starting with 05225f741fe8025afd6eb6fc27c0255fb0164b5231281e4624fe3dbc172bb17b not found: ID does not exist"
Oct 30 22:53:28.072338 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:53:28.072314    5061 scope.go:122] "RemoveContainer" containerID="5bfcc39a2867af7ef9b9abd6f4d69835a5faabb49c869c7c1651d3a169555dad"
Oct 30 22:53:28.072556 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:53:28.072514    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bfcc39a2867af7ef9b9abd6f4d69835a5faabb49c869c7c1651d3a169555dad\": container with ID starting with 5bfcc39a2867af7ef9b9abd6f4d69835a5faabb49c869c7c1651d3a169555dad not found: ID does not exist" containerID="5bfcc39a2867af7ef9b9abd6f4d69835a5faabb49c869c7c1651d3a169555dad"
Oct 30 22:53:28.072619 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:53:28.072563    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bfcc39a2867af7ef9b9abd6f4d69835a5faabb49c869c7c1651d3a169555dad"} err="failed to get container status \"5bfcc39a2867af7ef9b9abd6f4d69835a5faabb49c869c7c1651d3a169555dad\": rpc error: code = NotFound desc = could not find container \"5bfcc39a2867af7ef9b9abd6f4d69835a5faabb49c869c7c1651d3a169555dad\": container with ID starting with 5bfcc39a2867af7ef9b9abd6f4d69835a5faabb49c869c7c1651d3a169555dad not found: ID does not exist"
Oct 30 22:53:28.173915 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:53:28.173856    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fe39287f-2cc6-4335-8df5-f74fa55515a1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe39287f-2cc6-4335-8df5-f74fa55515a1-istio-podinfo\") pod \"fe39287f-2cc6-4335-8df5-f74fa55515a1\" (UID: \"fe39287f-2cc6-4335-8df5-f74fa55515a1\") "
Oct 30 22:53:28.173915 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:53:28.173887    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe39287f-2cc6-4335-8df5-f74fa55515a1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe39287f-2cc6-4335-8df5-f74fa55515a1-istiod-ca-cert\") pod \"fe39287f-2cc6-4335-8df5-f74fa55515a1\" (UID: \"fe39287f-2cc6-4335-8df5-f74fa55515a1\") "
Oct 30 22:53:28.173915 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:53:28.173906    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe39287f-2cc6-4335-8df5-f74fa55515a1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe39287f-2cc6-4335-8df5-f74fa55515a1-istio-envoy\") pod \"fe39287f-2cc6-4335-8df5-f74fa55515a1\" (UID: \"fe39287f-2cc6-4335-8df5-f74fa55515a1\") "
Oct 30 22:53:28.174165 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:53:28.173961    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe39287f-2cc6-4335-8df5-f74fa55515a1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe39287f-2cc6-4335-8df5-f74fa55515a1-credential-socket\") pod \"fe39287f-2cc6-4335-8df5-f74fa55515a1\" (UID: \"fe39287f-2cc6-4335-8df5-f74fa55515a1\") "
Oct 30 22:53:28.174165 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:53:28.174012    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe39287f-2cc6-4335-8df5-f74fa55515a1-kube-api-access-hkvx2\" (UniqueName: \"kubernetes.io/projected/fe39287f-2cc6-4335-8df5-f74fa55515a1-kube-api-access-hkvx2\") pod \"fe39287f-2cc6-4335-8df5-f74fa55515a1\" (UID: \"fe39287f-2cc6-4335-8df5-f74fa55515a1\") "
Oct 30 22:53:28.174165 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:53:28.174049    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe39287f-2cc6-4335-8df5-f74fa55515a1-istio-token\" (UniqueName: \"kubernetes.io/projected/fe39287f-2cc6-4335-8df5-f74fa55515a1-istio-token\") pod \"fe39287f-2cc6-4335-8df5-f74fa55515a1\" (UID: \"fe39287f-2cc6-4335-8df5-f74fa55515a1\") "
Oct 30 22:53:28.174165 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:53:28.174088    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe39287f-2cc6-4335-8df5-f74fa55515a1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe39287f-2cc6-4335-8df5-f74fa55515a1-istio-data\") pod \"fe39287f-2cc6-4335-8df5-f74fa55515a1\" (UID: \"fe39287f-2cc6-4335-8df5-f74fa55515a1\") "
Oct 30 22:53:28.174165 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:53:28.174117    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe39287f-2cc6-4335-8df5-f74fa55515a1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe39287f-2cc6-4335-8df5-f74fa55515a1-workload-socket\") pod \"fe39287f-2cc6-4335-8df5-f74fa55515a1\" (UID: \"fe39287f-2cc6-4335-8df5-f74fa55515a1\") "
Oct 30 22:53:28.174470 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:53:28.174224    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe39287f-2cc6-4335-8df5-f74fa55515a1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe39287f-2cc6-4335-8df5-f74fa55515a1-workload-certs\") pod \"fe39287f-2cc6-4335-8df5-f74fa55515a1\" (UID: \"fe39287f-2cc6-4335-8df5-f74fa55515a1\") "
Oct 30 22:53:28.174470 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:53:28.174335    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe39287f-2cc6-4335-8df5-f74fa55515a1-istiod-ca-cert" pod "fe39287f-2cc6-4335-8df5-f74fa55515a1" (UID: "fe39287f-2cc6-4335-8df5-f74fa55515a1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:53:28.174712 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:53:28.174621    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe39287f-2cc6-4335-8df5-f74fa55515a1-workload-socket" pod "fe39287f-2cc6-4335-8df5-f74fa55515a1" (UID: "fe39287f-2cc6-4335-8df5-f74fa55515a1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:53:28.174712 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:53:28.174631    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe39287f-2cc6-4335-8df5-f74fa55515a1-istio-data" pod "fe39287f-2cc6-4335-8df5-f74fa55515a1" (UID: "fe39287f-2cc6-4335-8df5-f74fa55515a1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:53:28.174712 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:53:28.174642    5061 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe39287f-2cc6-4335-8df5-f74fa55515a1-istiod-ca-cert\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:53:28.174712 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:53:28.174679    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe39287f-2cc6-4335-8df5-f74fa55515a1-credential-socket" pod "fe39287f-2cc6-4335-8df5-f74fa55515a1" (UID: "fe39287f-2cc6-4335-8df5-f74fa55515a1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:53:28.175017 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:53:28.174933    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe39287f-2cc6-4335-8df5-f74fa55515a1-workload-certs" pod "fe39287f-2cc6-4335-8df5-f74fa55515a1" (UID: "fe39287f-2cc6-4335-8df5-f74fa55515a1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:53:28.176152 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:53:28.176129    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fe39287f-2cc6-4335-8df5-f74fa55515a1-istio-podinfo" pod "fe39287f-2cc6-4335-8df5-f74fa55515a1" (UID: "fe39287f-2cc6-4335-8df5-f74fa55515a1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:53:28.176326 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:53:28.176307    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe39287f-2cc6-4335-8df5-f74fa55515a1-istio-envoy" pod "fe39287f-2cc6-4335-8df5-f74fa55515a1" (UID: "fe39287f-2cc6-4335-8df5-f74fa55515a1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:53:28.176564 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:53:28.176522    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe39287f-2cc6-4335-8df5-f74fa55515a1-kube-api-access-hkvx2" pod "fe39287f-2cc6-4335-8df5-f74fa55515a1" (UID: "fe39287f-2cc6-4335-8df5-f74fa55515a1"). InnerVolumeSpecName "kube-api-access-hkvx2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:53:28.176687 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:53:28.176668    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe39287f-2cc6-4335-8df5-f74fa55515a1-istio-token" pod "fe39287f-2cc6-4335-8df5-f74fa55515a1" (UID: "fe39287f-2cc6-4335-8df5-f74fa55515a1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:53:28.275446 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:53:28.275414    5061 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe39287f-2cc6-4335-8df5-f74fa55515a1-workload-certs\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:53:28.275446 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:53:28.275444    5061 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe39287f-2cc6-4335-8df5-f74fa55515a1-istio-podinfo\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:53:28.275446 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:53:28.275453    5061 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe39287f-2cc6-4335-8df5-f74fa55515a1-istio-envoy\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:53:28.275701 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:53:28.275462    5061 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe39287f-2cc6-4335-8df5-f74fa55515a1-credential-socket\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:53:28.275701 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:53:28.275471    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hkvx2\" (UniqueName: \"kubernetes.io/projected/fe39287f-2cc6-4335-8df5-f74fa55515a1-kube-api-access-hkvx2\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:53:28.275701 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:53:28.275482    5061 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe39287f-2cc6-4335-8df5-f74fa55515a1-istio-token\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:53:28.275701 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:53:28.275490    5061 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe39287f-2cc6-4335-8df5-f74fa55515a1-istio-data\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:53:28.275701 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:53:28.275499    5061 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe39287f-2cc6-4335-8df5-f74fa55515a1-workload-socket\") on node \"ip-10-0-130-60.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:53:28.470800 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:53:28.470731    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-8a1d54-openshift-default-6f54ddfbb5-rx7tn"]
Oct 30 22:53:28.474828 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:53:28.474801    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-authcfg-http-8a1d54-openshift-default-6f54ddfbb5-rx7tn"]
Oct 30 22:53:28.527176 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:53:28.527149    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe39287f-2cc6-4335-8df5-f74fa55515a1" path="/var/lib/kubelet/pods/fe39287f-2cc6-4335-8df5-f74fa55515a1/volumes"
Oct 30 22:53:30.111096 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:53:30.111063    5061 scope.go:122] "RemoveContainer" containerID="4ba0a33a5874f6cfeb648631d1628279b99f53722dba4e657e7979a188ef6b39"
Oct 30 22:53:30.119909 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:53:30.119889    5061 scope.go:122] "RemoveContainer" containerID="f9c4520b670857a0c392316c8e05950ef322223906e91407df16948e7ddaf64f"
Oct 30 22:53:54.669678 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:53:54.669595    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 22:54:24.665810 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:54:24.665771    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 22:54:35.142510 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:54:35.142480    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-f7b7d88f4-65wcn_2c715ba0-1ad7-4ae9-a88f-9261fed32e5b/authorino/0.log"
Oct 30 22:54:49.436943 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:54:49.436913    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-5fd44bb67c-4fxkc_08901302-db0c-4cd0-8ab8-d9d43e0c0c6b/maas-api/0.log"
Oct 30 22:54:52.157452 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:54:52.157408    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-f7b7d88f4-65wcn_2c715ba0-1ad7-4ae9-a88f-9261fed32e5b/authorino/0.log"
Oct 30 22:54:52.610541 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:54:52.610509    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-l2gwp_8e0f4d14-ce8b-42a3-8b8e-39db40615683/manager/0.log"
Oct 30 22:54:52.852577 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:54:52.852550    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-bm9ls_902c9b85-ffd2-47c1-abc0-d7a30633a3ad/manager/0.log"
Oct 30 22:54:53.839184 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:54:53.839149    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-j77xb_743ad69b-c9fa-4fb5-b942-baa0c949a7af/manager/0.log"
Oct 30 22:54:54.658851 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:54:54.658825    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-c4c65fbf-25bmv_db9fd2b8-dcbf-4145-9d53-6f030812a88d/istio-proxy/0.log"
Oct 30 22:54:54.666782 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:54:54.666748    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 22:54:56.117003 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:54:56.116969    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-694db55c87-7d2bf_e879c14b-842a-49da-a454-a85d9429b765/istio-proxy/0.log"
Oct 30 22:54:56.621116 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:54:56.621086    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-694db55c87-8pzt2_b3808649-2258-4508-be89-8e0c57ca83e2/istio-proxy/0.log"
Oct 30 22:54:56.845592 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:54:56.845558    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-694db55c87-8t2sw_d079dbb3-5d17-4dc9-9cf2-f07629b9f1f2/istio-proxy/0.log"
Oct 30 22:54:57.632488 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:54:57.632449    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-694db55c87-nbmc8_a77dd560-9aff-4292-b430-fa9c72d8c053/istio-proxy/0.log"
Oct 30 22:54:58.191088 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:54:58.191060    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-856cdd999f-4lt8x_36808b98-fda2-40eb-86a0-923a27e268ac/istio-proxy/0.log"
Oct 30 22:54:58.661068 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:54:58.661038    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-cxzp4_8867dea6-7639-4fca-95d7-ee7789e21a1d/payload-processing/0.log"
Oct 30 22:54:58.905008 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:54:58.904979    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6c7f96b45-4wj99_9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d/router/0.log"
Oct 30 22:54:59.775872 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:54:59.775836    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-zng7g_dc59d0e7-fbca-4882-8dda-f3f2095c7733/main/0.log"
Oct 30 22:55:00.511474 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:55:00.511443    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-ggrcq_cd4aa9fd-71de-4b7c-8c45-d7e41cf0ba9b/main/0.log"
Oct 30 22:55:01.017143 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:55:01.017112    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-hfpzd_8f552c94-29fe-485b-9410-5419467dd6ac/main/0.log"
Oct 30 22:55:01.466271 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:55:01.466240    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-blrrb_cb4b1ed5-2192-4877-8be7-ca82af831af5/main/0.log"
Oct 30 22:55:02.460049 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:55:02.460014    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-f7b7d88f4-65wcn_2c715ba0-1ad7-4ae9-a88f-9261fed32e5b/authorino/0.log"
Oct 30 22:55:14.569631 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:55:14.569550    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-5fd44bb67c-4fxkc_08901302-db0c-4cd0-8ab8-d9d43e0c0c6b/maas-api/0.log"
Oct 30 22:55:16.999035 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:55:16.999003    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-f7b7d88f4-65wcn_2c715ba0-1ad7-4ae9-a88f-9261fed32e5b/authorino/0.log"
Oct 30 22:55:17.441046 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:55:17.441011    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-l2gwp_8e0f4d14-ce8b-42a3-8b8e-39db40615683/manager/0.log"
Oct 30 22:55:17.675679 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:55:17.675650    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-bm9ls_902c9b85-ffd2-47c1-abc0-d7a30633a3ad/manager/0.log"
Oct 30 22:55:18.662635 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:55:18.662600    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-j77xb_743ad69b-c9fa-4fb5-b942-baa0c949a7af/manager/0.log"
Oct 30 22:55:19.296727 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:55:19.296693    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-c4c65fbf-25bmv_db9fd2b8-dcbf-4145-9d53-6f030812a88d/istio-proxy/0.log"
Oct 30 22:55:20.692657 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:55:20.692625    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-694db55c87-7d2bf_e879c14b-842a-49da-a454-a85d9429b765/istio-proxy/0.log"
Oct 30 22:55:21.185586 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:55:21.185552    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-694db55c87-8pzt2_b3808649-2258-4508-be89-8e0c57ca83e2/istio-proxy/0.log"
Oct 30 22:55:21.412243 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:55:21.412215    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-694db55c87-8t2sw_d079dbb3-5d17-4dc9-9cf2-f07629b9f1f2/istio-proxy/0.log"
Oct 30 22:55:22.196046 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:55:22.196016    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-694db55c87-nbmc8_a77dd560-9aff-4292-b430-fa9c72d8c053/istio-proxy/0.log"
Oct 30 22:55:22.755950 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:55:22.755925    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-856cdd999f-4lt8x_36808b98-fda2-40eb-86a0-923a27e268ac/istio-proxy/0.log"
Oct 30 22:55:23.242688 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:55:23.242656    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-cxzp4_8867dea6-7639-4fca-95d7-ee7789e21a1d/payload-processing/0.log"
Oct 30 22:55:23.465857 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:55:23.465828    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6c7f96b45-4wj99_9f96d744-56f0-4df2-a9a5-2a6bda6d8d0d/router/0.log"
Oct 30 22:55:24.066871 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:55:24.066843    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-zng7g_dc59d0e7-fbca-4882-8dda-f3f2095c7733/main/0.log"
Oct 30 22:55:24.665787 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:55:24.665755    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 22:55:24.738117 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:55:24.738086    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-ggrcq_cd4aa9fd-71de-4b7c-8c45-d7e41cf0ba9b/main/0.log"
Oct 30 22:55:25.185684 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:55:25.185658    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-hfpzd_8f552c94-29fe-485b-9410-5419467dd6ac/main/0.log"
Oct 30 22:55:25.627479 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:55:25.627446    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-blrrb_cb4b1ed5-2192-4877-8be7-ca82af831af5/main/0.log"
Oct 30 22:55:54.665561 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:55:54.665505    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 22:55:55.509321 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:55:55.509291    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-cxzp4_8867dea6-7639-4fca-95d7-ee7789e21a1d/payload-processing/0.log"
Oct 30 22:55:55.736198 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:55:55.736166    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-cxzp4_8867dea6-7639-4fca-95d7-ee7789e21a1d/payload-processing/0.log"
Oct 30 22:56:24.665632 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:56:24.665591    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 22:56:33.887068 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:56:33.887042    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-cxzp4_8867dea6-7639-4fca-95d7-ee7789e21a1d/payload-processing/0.log"
Oct 30 22:56:34.106918 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:56:34.106887    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-cxzp4_8867dea6-7639-4fca-95d7-ee7789e21a1d/payload-processing/0.log"
Oct 30 22:56:34.454898 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:56:34.454866    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-gjhpl/must-gather-fdd9g"]
Oct 30 22:56:34.455271 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:56:34.455259    5061 cpu_manager.go:414] "containerMap: removing container" podUID="fe39287f-2cc6-4335-8df5-f74fa55515a1" containerName="istio-proxy"
Oct 30 22:56:34.455318 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:56:34.455272    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe39287f-2cc6-4335-8df5-f74fa55515a1" containerName="istio-proxy"
Oct 30 22:56:34.455318 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:56:34.455287    5061 cpu_manager.go:414] "containerMap: removing container" podUID="fe39287f-2cc6-4335-8df5-f74fa55515a1" containerName="istio-proxy"
Oct 30 22:56:34.455318 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:56:34.455294    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe39287f-2cc6-4335-8df5-f74fa55515a1" containerName="istio-proxy"
Oct 30 22:56:34.455425 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:56:34.455374    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe39287f-2cc6-4335-8df5-f74fa55515a1" containerName="istio-proxy"
Oct 30 22:56:34.455425 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:56:34.455383    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe39287f-2cc6-4335-8df5-f74fa55515a1" containerName="istio-proxy"
Oct 30 22:56:34.458696 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:56:34.458676    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-gjhpl/must-gather-fdd9g"
Oct 30 22:56:34.510976 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:56:34.510950    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wxzs\" (UniqueName: \"kubernetes.io/projected/5746b952-100d-41fc-aca1-8b783806afe4-kube-api-access-5wxzs\") pod \"must-gather-fdd9g\" (UID: \"5746b952-100d-41fc-aca1-8b783806afe4\") " pod="openshift-must-gather-gjhpl/must-gather-fdd9g"
Oct 30 22:56:34.511096 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:56:34.511006    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/5746b952-100d-41fc-aca1-8b783806afe4-must-gather-output\") pod \"must-gather-fdd9g\" (UID: \"5746b952-100d-41fc-aca1-8b783806afe4\") " pod="openshift-must-gather-gjhpl/must-gather-fdd9g"
Oct 30 22:56:34.554514 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:56:34.554476    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-gjhpl\"/\"kube-root-ca.crt\""
Oct 30 22:56:34.554514 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:56:34.554505    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-gjhpl\"/\"default-dockercfg-mpw92\""
Oct 30 22:56:34.554762 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:56:34.554476    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-gjhpl\"/\"openshift-service-ca.crt\""
Oct 30 22:56:34.556577 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:56:34.556554    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-gjhpl/must-gather-fdd9g"]
Oct 30 22:56:34.611519 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:56:34.611494    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/5746b952-100d-41fc-aca1-8b783806afe4-must-gather-output\") pod \"must-gather-fdd9g\" (UID: \"5746b952-100d-41fc-aca1-8b783806afe4\") " pod="openshift-must-gather-gjhpl/must-gather-fdd9g"
Oct 30 22:56:34.611844 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:56:34.611821    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wxzs\" (UniqueName: \"kubernetes.io/projected/5746b952-100d-41fc-aca1-8b783806afe4-kube-api-access-5wxzs\") pod \"must-gather-fdd9g\" (UID: \"5746b952-100d-41fc-aca1-8b783806afe4\") " pod="openshift-must-gather-gjhpl/must-gather-fdd9g"
Oct 30 22:56:34.611844 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:56:34.611834    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/5746b952-100d-41fc-aca1-8b783806afe4-must-gather-output\") pod \"must-gather-fdd9g\" (UID: \"5746b952-100d-41fc-aca1-8b783806afe4\") " pod="openshift-must-gather-gjhpl/must-gather-fdd9g"
Oct 30 22:56:34.665262 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:56:34.665231    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wxzs\" (UniqueName: \"kubernetes.io/projected/5746b952-100d-41fc-aca1-8b783806afe4-kube-api-access-5wxzs\") pod \"must-gather-fdd9g\" (UID: \"5746b952-100d-41fc-aca1-8b783806afe4\") " pod="openshift-must-gather-gjhpl/must-gather-fdd9g"
Oct 30 22:56:34.773117 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:56:34.773059    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-gjhpl/must-gather-fdd9g"
Oct 30 22:56:35.143229 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:56:35.143195    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-gjhpl/must-gather-fdd9g"]
Oct 30 22:56:35.149480 ip-10-0-130-60 kubenswrapper[5061]: W1030 22:56:35.149454    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5746b952_100d_41fc_aca1_8b783806afe4.slice/crio-1c984f6b29589db0a481c49c19818f1d8ea5f1712dd6fade541c9ae188380346 WatchSource:0}: Error finding container 1c984f6b29589db0a481c49c19818f1d8ea5f1712dd6fade541c9ae188380346: Status 404 returned error can't find the container with id 1c984f6b29589db0a481c49c19818f1d8ea5f1712dd6fade541c9ae188380346
Oct 30 22:56:35.151316 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:56:35.151295    5061 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 22:56:36.069066 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:56:36.069029    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-gjhpl/must-gather-fdd9g" event={"ID":"5746b952-100d-41fc-aca1-8b783806afe4","Type":"ContainerStarted","Data":"1c984f6b29589db0a481c49c19818f1d8ea5f1712dd6fade541c9ae188380346"}
Oct 30 22:56:37.076015 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:56:37.075979    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-gjhpl/must-gather-fdd9g" event={"ID":"5746b952-100d-41fc-aca1-8b783806afe4","Type":"ContainerStarted","Data":"9155e9b5133085800900bcd47a33bfd073ee8304cbeab969e92d9866b5d37532"}
Oct 30 22:56:37.076408 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:56:37.076037    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-gjhpl/must-gather-fdd9g" event={"ID":"5746b952-100d-41fc-aca1-8b783806afe4","Type":"ContainerStarted","Data":"84ec1c99e6761c399c38db3b1257e1342d1cb0e93d526b8a8f6d8c1b32791f82"}
Oct 30 22:56:37.185589 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:56:37.185521    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-gjhpl/must-gather-fdd9g" podStartSLOduration=1.649048853 podStartE2EDuration="3.185507779s" totalImagesPullingTime="1.536458926s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:56:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:56:37.183765753 +0000 UTC m=+4450.796535509" watchObservedRunningTime="2026-10-30 22:56:37.185507779 +0000 UTC m=+4450.798277563"
Oct 30 22:56:43.419377 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:56:43.419290    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-6wbw4_fe41c70b-7e2c-432f-8389-0282d6fdb332/global-pull-secret-syncer/0.log"
Oct 30 22:56:45.121017 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:56:45.118590    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-bgnkz_3f97ee34-5dfc-454e-ba5c-afd939f3bd25/connection-checker/0.log"
Oct 30 22:56:45.531652 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:56:45.530658    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-cklcw_a32aacdc-53be-41cf-839e-435d43361890/connection-checker/0.log"
Oct 30 22:56:45.939502 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:56:45.939453    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-kz6dl_c798fd57-e0be-4b61-860b-0b833aca7784/connection-checker/0.log"
Oct 30 22:56:46.853550 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:56:46.849829    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-sztmx_238ff7d8-7da8-4cd8-9d45-deedf615211e/konnectivity-agent/0.log"
Oct 30 22:56:47.786492 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:56:47.786435    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-60.us-west-2.compute.internal_c0b65115e7be9fc3f67a9c81d1477e73/haproxy/0.log"
Oct 30 22:56:52.926240 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:56:52.926200    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-f7b7d88f4-65wcn_2c715ba0-1ad7-4ae9-a88f-9261fed32e5b/authorino/0.log"
Oct 30 22:56:53.232087 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:56:53.232013    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-l2gwp_8e0f4d14-ce8b-42a3-8b8e-39db40615683/manager/0.log"
Oct 30 22:56:53.628995 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:56:53.628963    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-bm9ls_902c9b85-ffd2-47c1-abc0-d7a30633a3ad/manager/0.log"
Oct 30 22:56:53.835250 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:56:53.835220    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-gjhpl/perf-node-gather-daemonset-d2wd5"]
Oct 30 22:56:53.842962 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:56:53.842935    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-gjhpl/perf-node-gather-daemonset-d2wd5"
Oct 30 22:56:53.895992 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:56:53.895901    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-gjhpl/perf-node-gather-daemonset-d2wd5"]
Oct 30 22:56:54.022894 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:56:54.022861    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f9789719-4274-4e99-8222-72159a486b5e-sys\") pod \"perf-node-gather-daemonset-d2wd5\" (UID: \"f9789719-4274-4e99-8222-72159a486b5e\") " pod="openshift-must-gather-gjhpl/perf-node-gather-daemonset-d2wd5"
Oct 30 22:56:54.023276 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:56:54.022930    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f9789719-4274-4e99-8222-72159a486b5e-podres\") pod \"perf-node-gather-daemonset-d2wd5\" (UID: \"f9789719-4274-4e99-8222-72159a486b5e\") " pod="openshift-must-gather-gjhpl/perf-node-gather-daemonset-d2wd5"
Oct 30 22:56:54.023276 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:56:54.022998    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjhq9\" (UniqueName: \"kubernetes.io/projected/f9789719-4274-4e99-8222-72159a486b5e-kube-api-access-cjhq9\") pod \"perf-node-gather-daemonset-d2wd5\" (UID: \"f9789719-4274-4e99-8222-72159a486b5e\") " pod="openshift-must-gather-gjhpl/perf-node-gather-daemonset-d2wd5"
Oct 30 22:56:54.023276 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:56:54.023029    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f9789719-4274-4e99-8222-72159a486b5e-proc\") pod \"perf-node-gather-daemonset-d2wd5\" (UID: \"f9789719-4274-4e99-8222-72159a486b5e\") " pod="openshift-must-gather-gjhpl/perf-node-gather-daemonset-d2wd5"
Oct 30 22:56:54.023276 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:56:54.023107    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f9789719-4274-4e99-8222-72159a486b5e-lib-modules\") pod \"perf-node-gather-daemonset-d2wd5\" (UID: \"f9789719-4274-4e99-8222-72159a486b5e\") " pod="openshift-must-gather-gjhpl/perf-node-gather-daemonset-d2wd5"
Oct 30 22:56:54.123725 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:56:54.123692    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f9789719-4274-4e99-8222-72159a486b5e-lib-modules\") pod \"perf-node-gather-daemonset-d2wd5\" (UID: \"f9789719-4274-4e99-8222-72159a486b5e\") " pod="openshift-must-gather-gjhpl/perf-node-gather-daemonset-d2wd5"
Oct 30 22:56:54.123884 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:56:54.123783    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f9789719-4274-4e99-8222-72159a486b5e-sys\") pod \"perf-node-gather-daemonset-d2wd5\" (UID: \"f9789719-4274-4e99-8222-72159a486b5e\") " pod="openshift-must-gather-gjhpl/perf-node-gather-daemonset-d2wd5"
Oct 30 22:56:54.123884 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:56:54.123839    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f9789719-4274-4e99-8222-72159a486b5e-podres\") pod \"perf-node-gather-daemonset-d2wd5\" (UID: \"f9789719-4274-4e99-8222-72159a486b5e\") " pod="openshift-must-gather-gjhpl/perf-node-gather-daemonset-d2wd5"
Oct 30 22:56:54.123884 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:56:54.123847    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f9789719-4274-4e99-8222-72159a486b5e-lib-modules\") pod \"perf-node-gather-daemonset-d2wd5\" (UID: \"f9789719-4274-4e99-8222-72159a486b5e\") " pod="openshift-must-gather-gjhpl/perf-node-gather-daemonset-d2wd5"
Oct 30 22:56:54.123884 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:56:54.123870    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjhq9\" (UniqueName: \"kubernetes.io/projected/f9789719-4274-4e99-8222-72159a486b5e-kube-api-access-cjhq9\") pod \"perf-node-gather-daemonset-d2wd5\" (UID: \"f9789719-4274-4e99-8222-72159a486b5e\") " pod="openshift-must-gather-gjhpl/perf-node-gather-daemonset-d2wd5"
Oct 30 22:56:54.124044 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:56:54.123893    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f9789719-4274-4e99-8222-72159a486b5e-proc\") pod \"perf-node-gather-daemonset-d2wd5\" (UID: \"f9789719-4274-4e99-8222-72159a486b5e\") " pod="openshift-must-gather-gjhpl/perf-node-gather-daemonset-d2wd5"
Oct 30 22:56:54.124044 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:56:54.123906    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f9789719-4274-4e99-8222-72159a486b5e-sys\") pod \"perf-node-gather-daemonset-d2wd5\" (UID: \"f9789719-4274-4e99-8222-72159a486b5e\") " pod="openshift-must-gather-gjhpl/perf-node-gather-daemonset-d2wd5"
Oct 30 22:56:54.124044 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:56:54.123925    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f9789719-4274-4e99-8222-72159a486b5e-podres\") pod \"perf-node-gather-daemonset-d2wd5\" (UID: \"f9789719-4274-4e99-8222-72159a486b5e\") " pod="openshift-must-gather-gjhpl/perf-node-gather-daemonset-d2wd5"
Oct 30 22:56:54.124044 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:56:54.124007    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f9789719-4274-4e99-8222-72159a486b5e-proc\") pod \"perf-node-gather-daemonset-d2wd5\" (UID: \"f9789719-4274-4e99-8222-72159a486b5e\") " pod="openshift-must-gather-gjhpl/perf-node-gather-daemonset-d2wd5"
Oct 30 22:56:54.179648 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:56:54.179571    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjhq9\" (UniqueName: \"kubernetes.io/projected/f9789719-4274-4e99-8222-72159a486b5e-kube-api-access-cjhq9\") pod \"perf-node-gather-daemonset-d2wd5\" (UID: \"f9789719-4274-4e99-8222-72159a486b5e\") " pod="openshift-must-gather-gjhpl/perf-node-gather-daemonset-d2wd5"
Oct 30 22:56:54.458014 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:56:54.456140    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-gjhpl/perf-node-gather-daemonset-d2wd5"
Oct 30 22:56:54.667661 ip-10-0-130-60 kubenswrapper[5061]: E1030 22:56:54.667625    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mstdj" containerName="extractor"
Oct 30 22:56:54.832106 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:56:54.832069    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-gjhpl/perf-node-gather-daemonset-d2wd5"]
Oct 30 22:56:54.838417 ip-10-0-130-60 kubenswrapper[5061]: W1030 22:56:54.838387    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podf9789719_4274_4e99_8222_72159a486b5e.slice/crio-eefca30626a901d59255af6b071bb89e92a16665d8c4f41b11be3dd5efebf2b8 WatchSource:0}: Error finding container eefca30626a901d59255af6b071bb89e92a16665d8c4f41b11be3dd5efebf2b8: Status 404 returned error can't find the container with id eefca30626a901d59255af6b071bb89e92a16665d8c4f41b11be3dd5efebf2b8
Oct 30 22:56:55.206367 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:56:55.206274    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-gjhpl/perf-node-gather-daemonset-d2wd5" event={"ID":"f9789719-4274-4e99-8222-72159a486b5e","Type":"ContainerStarted","Data":"d83931cb625e82ac5ff7bf920f6308d650973eafb84393b7667a23772732b832"}
Oct 30 22:56:55.206367 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:56:55.206343    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-gjhpl/perf-node-gather-daemonset-d2wd5" event={"ID":"f9789719-4274-4e99-8222-72159a486b5e","Type":"ContainerStarted","Data":"eefca30626a901d59255af6b071bb89e92a16665d8c4f41b11be3dd5efebf2b8"}
Oct 30 22:56:55.206908 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:56:55.206470    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-gjhpl/perf-node-gather-daemonset-d2wd5"
Oct 30 22:56:55.316811 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:56:55.316755    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-gjhpl/perf-node-gather-daemonset-d2wd5" podStartSLOduration=2.316732544 podStartE2EDuration="2.316732544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 22:56:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:56:55.316442771 +0000 UTC m=+4468.929212528" watchObservedRunningTime="2026-10-30 22:56:55.316732544 +0000 UTC m=+4468.929502301"
Oct 30 22:56:55.319170 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:56:55.319130    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-j77xb_743ad69b-c9fa-4fb5-b942-baa0c949a7af/manager/0.log"
Oct 30 22:57:00.578891 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:57:00.578855    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-ns759_102b8049-7349-4e2b-974a-44c901eebcf3/cluster-monitoring-operator/0.log"
Oct 30 22:57:01.222132 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:57:01.222100    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-gjhpl/perf-node-gather-daemonset-d2wd5"
Oct 30 22:57:05.475998 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:57:05.475962    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pqv7p_0090a21f-28fc-4b66-aef7-e4a23e118236/node-exporter/0.log"
Oct 30 22:57:05.870599 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:57:05.870573    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pqv7p_0090a21f-28fc-4b66-aef7-e4a23e118236/kube-rbac-proxy/0.log"
Oct 30 22:57:06.269840 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:57:06.269759    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pqv7p_0090a21f-28fc-4b66-aef7-e4a23e118236/init-textfile/0.log"
Oct 30 22:57:11.508106 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:57:11.508067    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-8pk6d_6339c2e5-a2c9-4e70-88e6-97285a01a8ff/prometheus-operator-admission-webhook/0.log"
Oct 30 22:57:16.083513 ip-10-0-130-60 kubenswrapper[5061]: I1030 22:57:16.083270    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-c4c65fbf-25bmv_db9fd2b8-dcbf-4145-9d53-6f030812a88d/istio-proxy/0.log"